Neuropong

This is the repo contains the NeuroPong project files. NeuroPong is an embedded, neuromorphic Atari (Pong) agent. It marries an event-based camera, TENNLab's Neuromorphic Starter Kit (https://arxiv.org/abs/2211.04526), and an Atari 2600 console. Spiking neural networks are trained on simulated events created from frames generated by the Arcade Learning Environment, and then, those spiking neural networks are put on hardware to examine how they perform in a live setting using real event camera inputs.

Prerequisites

Repositories needed to run existing spiking neural network agents:

TENNLab Open-Source Framework: https://github.com/TENNLab-UTK/framework-open
Inivation dv-processing Library: https://gitlab.com/inivation/dv/dv-processing
TENNLab Embedded Neuromorphic repository(*):
- Needed to compile code for Neuromorphic Starter Kit's Raspberry Pi Picos

(*) You will need to get access to this repo from Jim Plank (jplank@utk.edu) until it is open-sourced.

You will also need Python and a shell of some kind (bash/zsh) on Linux or Mac. It is recommended to set up a python virtual environment (Python <= 3.9) for compatability with Inivation event camera library.

/models: STL models of 3D printed objects used in this project for the presented prototype
/kit-code: Contains code used by embedded-neuromorphic/ repo to compile kit code.
/pcb: Repo containing KiCad files for the PCB that contains the Atari Translator circuit
/scripts: Scripts used to assist with flashing Picos and compiling kit code. calibration.py and event_server.py are used on the Raspberry Pi (or whatever is communicating with the Neuromorphic Starter Kit)
/imgs: Contains various diagrams and images detailing parts of the NeuroPong system
/networks: Contains the networks generated for the Neuropong paper.

Table of Hardware Materials/Components

	Part	Quantity	Cost
	PCB Design	1	$4.11
	AD5242	1	$4.55
	ULN2008	1 (unused)	$3.95
	Female pin strips	2	$26.00
	Raspberry Pi 4	1	$75.00
	MicroSD Card	1	$13.50
	Breakout Cable	1	$20.00
Neuromorphic Starter Kit
	PCB Design	1	$10.65
	Raspberry Pi Pico	3	$15.00
	Power Supply	1	$7.69
Total			$180.45

The Atari 2600 console and VideoOlympics cartridge together were ~$20.00.
The DAVIS346 camera used for this project was purchased from [Inivation] (https://shop.inivation.com/).

Running the Hardware Setup

Turn on monitor, put VideoOlympics cartridge in Atari 2600 console, and turn on the Atari console.
- You should see the Pong game appear on the screen.
Plug NeuroPong controller into the back of one of the Atari controller ports.
- NOTE: you will need to make the NeuroPong cable. An detailed construction diagram can be found in the /imgs folder
Power on the Raspberry Pi, and plug DAVIS346 camera into one of the USB ports on the Raspberry Pi.
Compile the network using app_compile from the embedded_neuromorphic/ repo
- Ex: ./bin/app_compile -j network_file -o kit-code/pico-pong_observations.c -a kit-code/pico-pong_actions[2,3].c -s
Flash the three Raspberry Pi Picos on the kit with the appropriate encoder, decoder, and network.
Power on the Neuromorphic Kit
On the Raspberry Pi, run scripts/calibration.py to assist with focusing the lens and cropping the camera's FOV to the screen.
- The output produced by scripts/calibration.py is the command that is used to run scripts/event_server.py.
  - It figures out what the crop coordinates are and what the downsampling parameters need to be to match whatever the target downsampled resolution is. Target resolution is specified via global variables in the scripts/calibration.py program.
Run the command output by scripts/calibration.py
- Ex: python3 event_server.py --crop ...
Wait 20-30 seconds for the system to initialize. If it's unreponsive, CTRL-C the event_server process and rerun it.
Begin the Atari game by toggling the ``Game Reset'' switch on the Atari system.

Running the networks in the `framework-open` processor_tool

You may want to run the networks using framework-open/bin/processor_tool in order to view spike rasters for a network at each timestep during an episode of that network playing Pong in the Arcade Learning Environment. In networks/processor_tool_commands are the commands for the processor tool, for each of the 32 networks, for one 2000 frame Pong episode each. Each file in networks/processor_tool_commands contains the Apply_Spike (AS) commands that communicate the event simulated and downsampled representation of each frame as they're generated by the ALE online. To run, do the following:

        ```
        UNIX> pwd
        .../neuropong
        UNIX> ./framework-open/bin/processor_tool_risp < networks/processor_tool_commands/pong0.txt
        0   INPUT  :    // No inputs are provided to the network during the first frame or timestep, so the spike raster is empty
        1   INPUT  : 
        2   INPUT  : 
        3   INPUT  : 
        4   INPUT  : 
        5   INPUT  : 
        6   INPUT  : 
        7   INPUT  : 
        8   INPUT  : 
        9   INPUT  : 
        10  INPUT  : 
        11  INPUT  : 
        12  INPUT  : 
        13  INPUT  : 
        14  INPUT  : 
        15  INPUT  : 
        16  INPUT  : 
        17  INPUT  : 
        18  INPUT  : 
        19  INPUT  : 
        20  INPUT  : 
        21  INPUT  : 
        22  INPUT  : 
        23  INPUT  : 
        24  INPUT  : 
        25  INPUT  : 
        26  INPUT  : 
        27  INPUT  : 
        28  INPUT  : 
        29  INPUT  : 
        30  INPUT  : 
        31  INPUT  : 
        32  INPUT  : 
        33  INPUT  : 
        34  INPUT  : 
        35  INPUT  : 
        36  INPUT  : 
        37  INPUT  : 
        38  INPUT  : 
        39  INPUT  : 
        40  INPUT  : 
        41  INPUT  : 
        42  INPUT  : 
        43  INPUT  : 
        44  INPUT  : 
        45  INPUT  : 
        46  INPUT  : 
        47  INPUT  : 
        48  INPUT  : 
        49  INPUT  : 
        50  INPUT  : 
        51  INPUT  : 
        52  INPUT  : 
        53  INPUT  : 
        54  INPUT  : 
        55  INPUT  : 
        56  INPUT  : 
        57  INPUT  : 
        58  INPUT  : 
        59  INPUT  : 
        60  INPUT  : 
        61  INPUT  : 
        62  INPUT  : 
        63  INPUT  : 
        64  INPUT  : 
        65  INPUT  : 
        66  INPUT  : 
        67  INPUT  : 
        68  INPUT  : 
        69  INPUT  : 
        70  INPUT  : 
        71  INPUT  : 
        72  INPUT  : 
        73  INPUT  : 
        74  INPUT  : 
        75  INPUT  : 
        76  INPUT  : 
        77  INPUT  : 
        78  INPUT  : 
        79  INPUT  : 
        80  INPUT  : 
        81  INPUT  : 
        82  INPUT  : 
        83  INPUT  : 
        84  INPUT  : 
        85  INPUT  : 
        86  INPUT  : 
        87  INPUT  : 
        88  INPUT  : 
        89  INPUT  : 
        90  INPUT  : 
        91  INPUT  : 
        92  INPUT  : 
        93  INPUT  : 
        94  INPUT  : 
        95  INPUT  : 
        96  INPUT  : 
        97  INPUT  : 
        98  INPUT  : 
        99  INPUT  : 
        100 INPUT  : 
        101 INPUT  : 
        102 INPUT  : 
        103 INPUT  : 
        104 INPUT  : 
        105 INPUT  : 
        106 INPUT  : 
        107 INPUT  : 
        108 INPUT  : 
        109 INPUT  : 
        110 INPUT  : 
        111 INPUT  : 
        112 INPUT  : 
        113 INPUT  : 
        114 INPUT  : 
        115 INPUT  : 
        116 INPUT  : 
        117 INPUT  : 
        118 INPUT  : 
        119 INPUT  : 
        120 INPUT  : 
        121 INPUT  : 
        122 INPUT  : 
        123 INPUT  : 
        124 INPUT  : 
        125 INPUT  : 
        126 INPUT  : 
        127 INPUT  : 
        128 INPUT  : 
        129 INPUT  : 
        130 INPUT  : 
        131 INPUT  : 
        132 INPUT  : 
        133 INPUT  : 
        134 INPUT  : 
        135 INPUT  : 
        136 INPUT  : 
        137 INPUT  : 
        138 INPUT  : 
        139 INPUT  : 
        140 INPUT  : 
        141 INPUT  : 
        142 INPUT  : 
        143 INPUT  : 
        144 INPUT  : 
        145 INPUT  : 
        146 INPUT  : 
        147 INPUT  : 
        148 INPUT  : 
        149 INPUT  : 
        150 INPUT  : 
        151 INPUT  : 
        152 INPUT  : 
        153 INPUT  : 
        154 INPUT  : 
        155 INPUT  : 
        156 INPUT  : 
        157 INPUT  : 
        158 INPUT  : 
        159 INPUT  : 
        160 OUTPUT : 
        161 OUTPUT : 
        162 OUTPUT : 
        0   INPUT  : 1000000000100100100000000000000000000000000000000 // Now, during the second frame/timestep, we apply a spike to  
        1   INPUT  : 1000000000000000000000000000000000000000000000000 // the appropriate input neurons and run the network for 50 cycles. 
        2   INPUT  : 1010000001000000010011000000000010000000000000000 // The columns represent the network timestep 
        3   INPUT  : 1000000000000000000000000000000000000000000000000 // and a "1" represents a neuronal fire.
        4   INPUT  : 1000000000000111100100100000001000100000000001000
        5   INPUT  : 1000000000000000000000000000000000000000000000000
        6   INPUT  : 1000000000000000000000000000000000000000000000000
        7   INPUT  : 1000000000000000000000000000000000000000000000000
        8   INPUT  : 1000000000000000000000000000000000000000000000000
        9   INPUT  : 1000000000000000000000000000000000000000000000000
        10  INPUT  : 1000000000000000000000000000000000000000000000000
        11  INPUT  : 1000000000000000000000000000000000000000000000000
        12  INPUT  : 1000000000000000000000000000000000000000000000000
        13  INPUT  : 1000000000000000000000000000000000000000000000000
        14  INPUT  : 1000000000000000000000000000000000000000000000000
        15  INPUT  : 1000000000000000000000000000000000000000000000000
        16  INPUT  : 1000000000100000000000000000000000000000000000000
        17  INPUT  : 1000000000000000000000000000000000000000000000000
        18  INPUT  : 1000000000000000000000000000000000000000000000000
        19  INPUT  : 1000000000000000000000000000000000000000000000000
        20  INPUT  : 1000000000000010000000000000000000000000000000000
        21  INPUT  : 1000000000000000000000000000000000000000000000000
        22  INPUT  : 1000000001000000000000000000000000000000000000000
        23  INPUT  : 1001000000000000000000000000000000000000000000000
        24  INPUT  : 1000000000000000000000000000000000000000000000000
        25  INPUT  : 1000000000000000000000000000000000000000000000000
        26  INPUT  : 1000000000000000000000000000000000000000000000000
        27  INPUT  : 1000010000000000000000000000000000000000000000000
        28  INPUT  : 1000000000000000000000000000000000000000000000000
        29  INPUT  : 1000000001010000000000000000000000000000000000000
        30  INPUT  : 1000000000000000000000000000000000000000000000000
        31  INPUT  : 1000000000000000000000000000000000000000000000000
        32  INPUT  : 1000000000000000000000000000000000000000000000000
        33  INPUT  : 1000000000000000000000000000000000000000000000000
        34  INPUT  : 1000000000000000000000000000000000000000000000000
        35  INPUT  : 1000000000000000000000000000000000000000000000000
        36  INPUT  : 1000000000000000000000000000000000000000000000000
        37  INPUT  : 1000000000000000000000000000000000000000000000000
        38  INPUT  : 1000000010000000000000000000000000000000000000000
        39  INPUT  : 1000000000000000000000000000000000000000000000000
        40  INPUT  : 1000000000010000000000000000000000000000000000000
        41  INPUT  : 1000000000000000000000000000000000000000000000000
        42  INPUT  : 1000000000000000000000000000000000000000000000000
        43  INPUT  : 1000000000000000000000000000000000000000000000000
        44  INPUT  : 1000000000000000000000000000000000000000000000000
        45  INPUT  : 1000000000000000000000000000000000000000000000000
        46  INPUT  : 1000000000000000000000000000000000000000000000000
        47  INPUT  : 1000000000000000000000000000000000000000000000000
        48  INPUT  : 1000000000010000000000000000000000000000000000000
        49  INPUT  : 1000000000000000000000000000000000000000000000000
        50  INPUT  : 1000000000010000000000000000000000000000000000000
        51  INPUT  : 1000000100000000000000000000000000000000000000000
        52  INPUT  : 1000000000000000000000000000000000000000000000000
        53  INPUT  : 1000000000000000000000000000000000000000000000000
        54  INPUT  : 1000000000000000000000000000000000000000000000000
        55  INPUT  : 1000000000000000000000000000000000000000000000000
        56  INPUT  : 1000000000000000000000000000000000000000000000000
        57  INPUT  : 1000000000000000000000000000000000000000000000000
        58  INPUT  : 1001010000000110000000000001111001001000000010001
        59  INPUT  : 1000100000000001000000000000000000000000000000000
        60  INPUT  : 1000000000000000000000000000000000000000000000000
        61  INPUT  : 1000000000010000000000000000000000000000000000000
        62  INPUT  : 1000000000000000000000000000000000000000000000000
        63  INPUT  : 1000000000000000000000000000000000000000000000000
        64  INPUT  : 1100010100100100000000000000000000000000000000000
        65  INPUT  : 1000000000000000000000000000000000000000000000000
        66  INPUT  : 1010010010000000000000000000000000000000000000000
        67  INPUT  : 1000000000010000000000000000000000000000000000000
        68  INPUT  : 1000000000000000000000000000000000000000000000000
        69  INPUT  : 1001000000000000000000000000000000000000000000000
        70  INPUT  : 1000010000000000000000000000000000000000000000000
        71  INPUT  : 1000000000000000000000000000000000000000000000000
        72  INPUT  : 1000000000000000000000000000000000000000000000000
        73  INPUT  : 1000000000000000000000000000000000000000000000000
        74  INPUT  : 1000000000000000000000000000000000000000000000000
        75  INPUT  : 1000000000000001000000000000000000000000000000000
        76  INPUT  : 1000000000000000000000000000000000000000000000000
        77  INPUT  : 1000000000000000000000000000000000000000000000000
        78  INPUT  : 1001011000100000000000000100000000000000000000000
        79  INPUT  : 1000000000000000000000000000000000000000000000000
        80  INPUT  : 1000000000000000000000000000000000000000000000000
        81  INPUT  : 1000000000000000000000000000000000000000000000000
        82  INPUT  : 1000000000000000000000000000000000000000000000000
        83  INPUT  : 1000000000000000000000000000000000000000000000000
        84  INPUT  : 1000000000000000000000000000000000000000000000000
        85  INPUT  : 1000000000000000000000000000000000000000000000000
        86  INPUT  : 1000000000000000000000000000000000000000000000000
        87  INPUT  : 1000000000000000000000000000000000000000000000000
        88  INPUT  : 1000000000000000000000000000000000000000000000000
        89  INPUT  : 1000000000000000000000000000000000000000000000000
        90  INPUT  : 1000000000000000000000000000000000000000000000000
        91  INPUT  : 1000000000000000000000000000000000000000000000000
        92  INPUT  : 1000000000000000000000000000000000000000000000000
        93  INPUT  : 1000000000000000000000000000000000000000000000000
        94  INPUT  : 1000000000000000000000000000000000000000000000000
        95  INPUT  : 1000000000000000000000000000000000000000000000000
        96  INPUT  : 1000000000100001000000000010000000000000000000000
        97  INPUT  : 1000000000000000000000000000000000000000000000000
        98  INPUT  : 1010000000000000000000000000000000000000000000000
        99  INPUT  : 1000000000000000000000000000000000000000000000000
        100 INPUT  : 1000000000000000000000000000000000000000000000000
        101 INPUT  : 1000000100000000000000000000000000000000000000000
        102 INPUT  : 1000000000000000000000000000000000000000000000000
        103 INPUT  : 1000000000000000000000000000000000000000000000000
        104 INPUT  : 1000000000000000000000000000000000000000000000000
        105 INPUT  : 1000000000000000000000000000000000000000000000000
        106 INPUT  : 1000000000000000000000000000000000000000000000000
        107 INPUT  : 1000000000000000000000000000000000000000000000000
        108 INPUT  : 1000000000000000000000000000000000000000000000000
        109 INPUT  : 1000000000000000000000000000000000000000000000000
        110 INPUT  : 1000000000000000000000000000000000000000000000000
        111 INPUT  : 1000000000000000000000000000000000000000000000000
        112 INPUT  : 1000000000000000000000000000000000000000000000000
        113 INPUT  : 1001001000000000000000000000000000000000000000000
        114 INPUT  : 1000000000000000000000000000000000000000000000000
        115 INPUT  : 1000000000000000000000000000000000000000000000000
        116 INPUT  : 1000000000000000000000000000000000000000000000000
        117 INPUT  : 1000000000000000000000000000000000000000000000000
        118 INPUT  : 1000000000100000000000010000000000100000000000000
        119 INPUT  : 1000000000000000000000000000000000000000000000000
        120 INPUT  : 1000000000000100000000001000000000000000000000000
        121 INPUT  : 1000000000000000000000000000000000000000000000000
        122 INPUT  : 1010000000000000010000000000000000000000000000000
        123 INPUT  : 1000000000000000000000000000000000000000000000000
        124 INPUT  : 1000000000000000000000000000000000000000000000000
        125 INPUT  : 1000000000000000000000000000000000000000000000000
        126 INPUT  : 1000000000000000000000000000000000000000000000000
        127 INPUT  : 1000000000000000000000000000000000000000000000000
        128 INPUT  : 1000000000000000000000000000000000000000000000000
        129 INPUT  : 1011000100000000000000100000000000000000000000000
        130 INPUT  : 1000000000000000000000000000000000000000000000000
        131 INPUT  : 1000000000000000000000000000000000000000000000000
        132 INPUT  : 1000000000000000000000000000000000000000000000000
        133 INPUT  : 1000000000000001000000000000000000000000000000000
        134 INPUT  : 1000000000000000000000000000000000000000000000000
        135 INPUT  : 1010000000000000000000000000000000000000000000000
        136 INPUT  : 1000000000000000000000000000000000000000000000000
        137 INPUT  : 1000000000000000000000000000000000000000000000000
        138 INPUT  : 1000000000000000000000000000000000000000000000000
        139 INPUT  : 1000000000000000000000000000000000000000000000000
        140 INPUT  : 1000000000000000000000000000000000000000000000000
        141 INPUT  : 1000000000000000000000000000000000000000000000000
        142 INPUT  : 1000000010000000000000000000000000000000000000000
        143 INPUT  : 1000000000000000000000000000000000000000000000000
        144 INPUT  : 1000000100000001000100000000001000000000000000000
        145 INPUT  : 1000000000000000000000000000000000000000000000000
        146 INPUT  : 1000000000000000000000000000000000000000000000000
        147 INPUT  : 1000000000000000000000000000000000000000000000000
        148 INPUT  : 1000100000000000000000000000000000000000000000000
        149 INPUT  : 1000000000000000000000000000000000000000000000000
        150 INPUT  : 1100010000000000000010000000000000000000000000000
        151 INPUT  : 1000000000000000000000000000000000000000000000000
        152 INPUT  : 1000000000000000000000000000000000000000000000000
        153 INPUT  : 1000000000000000000000000000000000000000000000000
        154 INPUT  : 1010000000000000000000000000000000000000000000000
        155 INPUT  : 1000000000000000000000000000000000000000000000000
        156 INPUT  : 1000000000000000000000000000000000000000000000000
        157 INPUT  : 1000000000000000000000000000000000000000000000000
        158 INPUT  : 1000000100000010000000000000000000000000000000000
        159 INPUT  : 1000000000000000000000000000000000000000000000000
        160 OUTPUT : 0000100101000000011000000000000111100100100000001 // At the end of our 50 timesteps, we count the output fires on the
        161 OUTPUT : 0000000010000000000000000000000000000000000000000 // three output neurons. Whoever fired the most "wins", and its
        162 OUTPUT : 0000000001000100000000000000000000000000000000000 // action is the action performed by the network for that frame
        ...                                                            // from the ALE.
        ...
        ```

More information about processor_tool_risp can be found in the framework-open/markdown/ directory. This is provided as a method for you to see how the network acts when provided event-simulated and downsampled ALE frames.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Neuropong

Prerequisites

Contents

Table of Hardware Materials/Components

Running the Hardware Setup

Running the networks in the `framework-open` processor_tool

About

Uh oh!

Releases

Packages

Uh oh!

Contributors 2

Uh oh!

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
imgs		imgs
kit-code		kit-code
models		models
networks		networks
pcb		pcb
scripts		scripts
LICENSE		LICENSE
README.md		README.md

License

TENNLab-UTK/neuropong

Folders and files

Latest commit

History

Repository files navigation

Neuropong

Prerequisites

Contents

Table of Hardware Materials/Components

Running the Hardware Setup

Running the networks in the framework-open processor_tool

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors 2

Uh oh!

Languages

Running the networks in the `framework-open` processor_tool

Packages