musicalGPT

A 15 minute song inside which an entire GPT model lives. This implementation is an added layer on Henry Zhang's implementation where he converted Andrej Karpathy's implementation of the GPT into a prime number. I took the decimal representation of the prime and converted it into a song. For this I am treating the entire prime number as a string and assigning the most frequent element as "Sa" and the second most frequent element as "Pa" which are the most common notes in Indian classical music. The third most frequent element is assigned as "Sa'" which is Sa of the next octave and to account for the extra 2 notes I have kept one note "Ga'" and "'Ni" and these are the least common elements in this.

These notes are taken as semitones from Sa and my Sa in this implementation is the one in harmonium.wav. The semitone for Sa is 0 and for the notes above it's a positive number while for the notes below it is a negative number.

Note	Digit	Semitones from Sa
Sa	3	0
Pa	0	+7
Sa'	9	+12
Re	2	+2
Ga	8	+4
Ma	5	+5
Komal Dha	4	+8
Ni	7	+11
Ga'	1	+16
'Ni	6	-1

To go from GPT to this prime, refer to the sections below which are taken as it is from Zhang's readme.

femtoGPT

A GPT model that is a prime number.

Yesterday, Kuber Mehta made a post about putting a GPT model inside a QR code.

What's the fun in a QR code?

I trimmed his code down to about 53 lines and made a prime number from it.

This repo contains a 3,000-digit prime whose bytes are a gzip stream that decompresses to a complete, trainable GPT implementation -- in pure Python with zero dependencies.

The Prime

Hexadecimal (click to expand)

0x1f8b080000000000021395566d6fdb3610fe1cfd0a21c01052a665cb4bba4e0407b445d2a44dfa969721d5048392285bb1de22cab6eaacff7d474ab29d60fb30c0b08e3c3d77bcbbe78e922c91492e6b9e87827e639ba4a4574c2e337ac9ca2ac96b7ac65291d3b7ace2f94c18495616556d169264bc9e9b5c9a9f0868a22253f2b5716d4b2122743cc146ca8e92bc5cd676ddd44746129b79a1807609405b3489ac254ab1db595c56699a0476251e9742d6f4f9d2866525ea2a112b818ee6755d4a7734aaf8da9e25f57c192ca5a8c222af455edb61918d16bc525e7e8c32be10595189512562399a0b1e49d893b5a84639cf84d4672329363e338fdb123c9408c745657233c9cda214391c114ec123847bb52dcb34a9d1d15ff91186a8b6305f053f5fc6712ad0674c2fd161becccca808a57b48ce600b1b374c42ac901f29c0c091fd5024b952607acbced00da677ec76e028e8aa087960ca6423007c878d30e5529aa7ae711089d89c4e933ca9a753b421110919c264067fd8ddd811af398be8c69e553c626310a6210bd563c6663d984791c616d82d586142101215e4148b540af314159842b297550e726b7150e807d118821ce240309db16c99fe3f63d67363ddaad5edcc96c55a9b0db0fb126f050a0bd0c0ea7750307430e9d16931439b3ddc275befb41e3aac336ad75b9068ca1720bdf30cf47caf4756225d3e8366bc4163f2c29f6d75c03fc62d743aad742158cab320e226a4c3dd0c0aadd0497da6b08a3e337219f409ef3c02ca1a3aad3ee0e162cdab481dc838380899e7d38829be814bfd468c565a7500a55ae9a604aa476e64c3694045bd1835ba071aa55801817c1adabc8466507a40c660bca798a3d6fdcb15f4672581df61eb41294a3253aa6fa8014b04fe66d82d3576c06656a3252367ce2b9ab1637acef2d128a3f7ecb5f1be8fbf200991cc1e4f5ccf3b45d7f68c2fa584044bac8f3955e6dfa204fb7bab02daf11df3dea33b9263f21eddb78f6e95ffc7838c95706c6d77943806537f322f34b773e19d160325722d864a0cfcbee74b5ef1acedfa3fa1d2177d200d59bbde15e256d0da22419b194e1abc9b3a6bdf5065faae8663c81497b8e6cdce7f8a69c03cc48721d684dc615355ec2b14f44de7712bb2801b3b6cd05aff02357639bcbab2565ab952ca065b67a05008f080f8c011c313b5b44f7a83abeef0edfbbef1a60f0ea271bd66f0436b1bf2a30f2dc0adc7afa82625f9483e286a34ec0b7a83de7963dfab7df2ce737caff4610cce584395aec1f4915da00654bffa982e3af918e455279f80fcb1a7e502d30f3b8ad20a586f68f6cd5b3a64d8156c6e9dd3357bf4842b06e73e5d4898f9dd629b9e8f3e5dfddbfe079f4ae8a52bb4f612df5a48efc10741a993d6c339c6d6b9ce944ec043bb7b8616c052c096ec3b92219c0c0a56ab535e21592a532b09ff606dcfd4195a0108ef993957f3a2616fd005aa20f4573e0cfc3e596a13ee2e3d84a054935da92f904a24bcff9b8f55925fb7b06e68b4599cf87051572470483021a2842e7388fd3bfc4e08d4fe35b0c9b3c7bea5884cc59e1cb393f1d8d0fddd1e315665ddb0cf5ef98bbaeb7c389d77eb0fbc1b3bc923011cde917403dbb73e2d5806b7df3d34c90cc3f4a6400d281b8179f5b82d5fd277b3fbd897f73bfa8a669033180a8a4cd883c5c0f17d3de33130175295165202b71fb155682eabb5bd9d8c98a6d584a595659f58c8197c824f0609a3bd4cac7214ab4c6bcf64ad7c0be86751f15a40d06e046e59e058ea3940ce3070b0b5d6038c0aad9a58a2574d7a159484ae75030fc1e7c4420a3dd268b8b7ca015ca5788490e87727db5d0b0e381025a6ebee36370e2e517c085f2fa5f904afb8c7d14f73643ec55af8db54519a4f3a56e5ceb58fe39f87d8b8549f2ac3e110c2894525e03bcf84d51136f6083619abf2ed2a50b3dbbd26ea4a7caf87fa02980c63658f67bbce8696acd9b51dce8b241412bd457798782fa7d7c2c7d0f786be7f6ac66edd00beaf16b0aefa12dfc050c05db03c2b53613e3d40b893e8a76b3ef59f4d1586e0fe013de57ab7be0a00000000139b

Decimal (click to expand)

256692052326461880979457083018264343196018398194332365911142308289967772861812037587566524307093408770939588252723599609612339236620322460355534096040734115212485952029673528850625727380336045612690969423469478707207634022980174259644704055667150997827839955733555320772117923546518952217300801249866673897550105064433010563643378766173550181752333987444385686415147603105955949169652448094114023206691727104887685488355788782538542317678495474155251982729082153107790518330582547925185808647745086620345030600203948254048228941075922929039093850696525128290705610292245931025163479091457080329225540437354980589396701755688073848555246962734706716110731841724071813679083872781357314558081500688415318185548361799412169198803579207646358980698892893584065537538324025901319801470622603428559074087891852118842965592211024758678389725644329281009970986080772567460888688341737506950206141202868152015892264391071480907364699290744036993948337479998712238358740237750880402941104287787740303417326216658799783251167074059038290835311752815289904327129233734415591614339027704875180882954302503246234265202817887367771045878780833911299874908342786212229403286116604160121614331398436335124879088206908735516125265267245400277749556448046591892240046159871024126444052677033724297045987447738668306084159285519600183731853325518220721194437789444101002328601064677295505699576133015245561779017682068094378769263019056769728539628575335236414438071536867577945983637233174695504684002932925051266209937731950947719328172727337333422860839634743499713086684409057959895150810742473869033362402113416786384256525219580223341508743864793999444702325632722262641001675423013389402312019376409099900954079928388766197273450091360122910687789021736819544331145761169187069188568788261363437091835129898255147709851699231380322837799959025189905223699251820454406668342440370560754513953857068401733395227754034466662571335143031797186130138643432941335774058324327160569291450613235007758168304407592372551156029911336934088460169538393422575280034107589322880051980211042188820225949466732357186402117841450996250104280823372434756129335528475031635847312741663500819631970637292418280174936200931894555797440949387960064288510857596412984533540855586822897859265889237741428082430630993872794540710195687849115231608255722029665133148963101645432658879576403732481897362120052017344855169328586425589868184772909030031726330273070093543986928920709085362565522423842856459723080390008731158514048157831154234376677244536016759026929629712810466952458933629245919595030901718995447059603794607837823368390213767679971450068339938573463085996221683458603037439344111781991777780979000445992407594889002478675373238270945597345916052243371405449856639339051728942857498069832818062759777464860037501019210138166842477463746637612107515386478082040164197131064332939064589898730608615248401266501849232323523436154981158050690960246415203427240408421917329342530835821730377111194187463341277791471341276878620167410395683623174784942167790890313546311500352117690138951352445400618950820630473981697954974981836004956925651664518907273204594231233181921406063657324939252988541429599658475989549515321701529648097602406683526114688249509008277495632052971815178213926585168423169953513389181581847295040731523094860166495992302515653371891655529970095385722732139897443020260236343171404989817534374933922949938629283713818583667697802439556215293767038471302194442138208250989459734256077905247333841706688195935940671631061676725024076589621074935583928243909941431729649380703056068699702005911476366218139

Quick start

To see femtoGPT from the prime number above

python verify_prime.py

To quickly run the model in the prime number

python3 -c "$(python3 verify_prime.py)"

This takes the prime number embedded in the script, converts it to bytes, decompresses via gzip, and prints a working GPT. Copy the output to a file and run it to train a character-level language model.

The pipeline

picoGPT.py    Kuber Mehta's pico GPT (https://github.com/Kuberwastaken/picogpt/blob/main/picogpt.py) 
       | golf
       v
femtoGPT.py      same model in ~50 lines
       | gzip compress + pad 4 bytes
       v
  big integer       interpret bytes as big-endian int
       | gmpy2.next_prime()
       v
  THE PRIME          5,942 digits, but a valid gzip stream

Since gzip decompressors stop reading after the first valid stream and ignore trailing bytes. Appending 4 bytes of padding after the compressed data gives enough room (~2^32 candidates) to find a nearby prime without disturbing the gzip payload.

This is the same technique Phil Carmody used in 2001 to construct an "illegal prime" encoding DeCSS. (https://en.wikipedia.org/wiki/Illegal_number)

Reproducing this

The prime-finding scripts require gmpy2 for fast primality testing:

pip install -r requirements.txt

python find_prime.py
python verify_prime.py