Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
432 commits
Select commit Hold shift + click to select a range
a9ac286
fix metadata (debugging)
Jakobeha Jul 6, 2023
54e07d1
src and pool idx metadata
Jakobeha Jul 6, 2023
dab80a5
debug source and constant pool metadata
Jakobeha Jul 6, 2023
7816a15
cppcheck + minor bugfix
Jakobeha Jul 7, 2023
b475c00
we only need to load source and constant pool indices when DEBUG_SERI…
Jakobeha Jul 7, 2023
f3797b0
print DispatchTable
Jakobeha Jul 7, 2023
f0ac353
Only hash baseline so the hash doesn't change
Jakobeha Jul 7, 2023
a96e491
add exit code to test failure message
Jakobeha Jul 7, 2023
93afb18
fix log issue
Jakobeha Jul 7, 2023
cd3de91
attempt to once again fix gitlab (probably not working)
Jakobeha Jul 7, 2023
e99dffe
add assertions
Jakobeha Jul 7, 2023
c0b1e39
more assertions
Jakobeha Jul 7, 2023
23db30f
fix possible OSR bug but probably intended; why does OSR call rir2pir…
Jakobeha Jul 7, 2023
bce1e70
compiler client/server test works now (fix patching function metadata)
Jakobeha Jul 7, 2023
6fb405f
replace stopifnot with warnifnot in test-compiler-client.r, because i…
Jakobeha Jul 7, 2023
c829cb1
fix patched string pointer
Jakobeha Jul 8, 2023
e599b08
Preserve DeoptMetadata
Jakobeha Jul 10, 2023
6aa9284
Disable OSR on compiler client, but provide the ability to force enable
Jakobeha Jul 10, 2023
a0ec89b
fix warnifnot in test-compiler-client.r
Jakobeha Jul 10, 2023
e42c5b0
fix minor CompilerClient memoize issue
Jakobeha Jul 10, 2023
4b4c4d3
fix actual warnifnot issue
Jakobeha Jul 10, 2023
4592156
try skipping less for hash, relax a lot of things...
Jakobeha Jul 10, 2023
9cb0963
don't redundantly add to the ConnectedWorklist
Jakobeha Jul 10, 2023
5c807c9
@WIP draft big and small hashes, but currently we put everything in t…
Jakobeha Jul 11, 2023
57c1cee
@WIP actually use big and small hashes
Jakobeha Jul 12, 2023
a2dedaf
@WIP update documentation
Jakobeha Jul 12, 2023
009943c
bugfixes
Jakobeha Jul 12, 2023
f4eeb88
don't copy refTable, instead we just always write to null stream...
Jakobeha Jul 13, 2023
dda6783
measure serialization and interning + generate report (draft)
Jakobeha Jul 13, 2023
2c78c5e
order timed events correctly
Jakobeha Jul 13, 2023
22ee7b1
fix unnecessary printing
Jakobeha Jul 14, 2023
2f0a2c7
more specific perf tests
Jakobeha Jul 14, 2023
e2e0bed
add extra pool to debug print
Jakobeha Jul 15, 2023
b530403
revert big and small UUID, but not measuring
Jakobeha Jul 18, 2023
9dbe061
measure time spent in compiler client and server (e.g. requests) too
Jakobeha Jul 18, 2023
139bd20
preserve objects in the connected worklist
Jakobeha Jul 18, 2023
fffe832
add reg-tests to compiler client
Jakobeha Jul 18, 2023
92c32d4
protect more SEXPs to attempt to fix gc issues
Jakobeha Jul 18, 2023
36659b5
try not hashing function flags so that we don't have to reintern
Jakobeha Jul 19, 2023
8d98f77
don't run code finalizer on exit
Jakobeha Jul 19, 2023
2972e0b
update measurement display to put important metrics at top + fix time…
Jakobeha Jul 19, 2023
a3b52ae
use xxHash instead of EVP
Jakobeha Jul 19, 2023
d5e9ecc
when measuring detailed event duration sums, don't count overlapping …
Jakobeha Jul 19, 2023
66e3845
increase protects and measure deserialization
Jakobeha Jul 20, 2023
8856ddf
@WIP draft alternative hash. TODO: rir hash impls
Jakobeha Jul 22, 2023
a31957b
@WIP draft alternative hash with rir hash impls
Jakobeha Jul 23, 2023
39db030
expose altrep methods used by serialize.c
Jakobeha Jul 23, 2023
74353b7
@WIP draft alternative hash with rir hash impls (progress...)
Jakobeha Jul 23, 2023
7da9b85
separate addConnected from hash and more progress
Jakobeha Jul 23, 2023
92b54c3
@WIP bugfixes...
Jakobeha Jul 23, 2023
dcdaf23
add and use peekLong + make Response an enum class
Jakobeha Jul 23, 2023
6f51d55
document more, like why server doesn't get retrieve requests from client
Jakobeha Jul 23, 2023
0afd7b5
don't hash printing Code src if we can, to improve debug perf
Jakobeha Jul 23, 2023
7142568
improve hashing ASTs (bring back SerialAst code), also fixes a bug in…
Jakobeha Jul 23, 2023
9a33b34
@WIP bugfixes
Jakobeha Jul 23, 2023
51fc933
measure hashing AST
Jakobeha Jul 23, 2023
df6b7f6
time server sending client new SEXP responses
Jakobeha Jul 23, 2023
e8caed6
fix compiler client/server nested timers, by adding support for neste…
Jakobeha Jul 23, 2023
a2a4df9
measure more fine-grained in hashing
Jakobeha Jul 23, 2023
8fbfd0e
@WIP bugfixes
Jakobeha Jul 24, 2023
8eee357
bugfix: return false when we re-intern a module but it was previously…
Jakobeha Jul 25, 2023
6c6a1b6
don't dump measure associated data or debug disassembly (put under co…
Jakobeha Jul 25, 2023
df03ef3
use different way to preserve SEXPs in UUIDPool to try and preserve t…
Jakobeha Jul 25, 2023
68ce1c6
don't initialize data in ArglistOrder.h because it fails to compile o…
Jakobeha Jul 25, 2023
2d400c8
more protect and preserve to try and ensure GC doesn't break the comp…
Jakobeha Jul 25, 2023
45b873e
@WIP bugfixes
Jakobeha Jul 25, 2023
c76b3aa
revert forcePreserve because R_PreserveObject does handle redundant c…
Jakobeha Jul 25, 2023
8d12c77
add EXTERNALSXP to CHKVEC in STRICT_TYPECHECK
Jakobeha Jul 25, 2023
1991788
update for enable-write-barrier
Jakobeha Jul 25, 2023
3e64538
make object decoded in LLVM modules actually gc-able
Jakobeha Jul 26, 2023
ef9a487
@WIP
Jakobeha Jul 26, 2023
4e7cd86
@WIP
Jakobeha Jul 26, 2023
f22d5b9
fix cppcheck
Jakobeha Jul 26, 2023
1ae1e9f
refactor: move hashing and serialization, use RirObjectPrintStyle (+ …
Jakobeha Jul 26, 2023
edb9e5c
add LazyArglist, LazyEnvironment, and PirTypeFeedback hash and addCon…
Jakobeha Jul 26, 2023
eb7d76f
add logging code because it wasn't in lasst commit due to a bad globa…
Jakobeha Jul 26, 2023
b6e6e70
@WIP
Jakobeha Jul 26, 2023
08e7948
add retrieve hash and read refs to other PIR types, and check other P…
Jakobeha Jul 27, 2023
9739559
transmit only function between client and server, not entire Dispatch…
Jakobeha Jul 27, 2023
cd1cb1a
also transmit oldOptFunction because PIR may use it
Jakobeha Jul 27, 2023
55297f1
@WIP print rir objects in fancy HTML graph
Jakobeha Jul 28, 2023
1fc3a6a
Added `PIR_PRINT_INTERNED_RIR_OBJECTS` and `PIR_PRINT_INTERNED_RIR_OB…
Jakobeha Jul 28, 2023
493573d
Separate old and pretty graph prints more, and more importantly, add …
Jakobeha Jul 29, 2023
82381e0
bugfixes
Jakobeha Jul 29, 2023
718b58b
improve cytoscape graph generation, generate less graphs, and bugfixes
Jakobeha Jul 29, 2023
7c7d86f
add proper arrows to code objects whose function bodies are different
Jakobeha Jul 29, 2023
f42b777
graph-print includes SEXP types, and Code sources expected
Jakobeha Jul 29, 2023
0fcd4f5
fixes for gcc
Jakobeha Jul 29, 2023
c555b75
obvious bugfix
Jakobeha Jul 29, 2023
1a35464
bugfixes
Jakobeha Jul 29, 2023
4733f3d
Remove unnecessary warning (we can't remove the unintern finalizer so…
Jakobeha Jul 30, 2023
d60cc18
graph print children of non-RIR objects which may be RIR objects
Jakobeha Jul 30, 2023
88850b0
fixes
Jakobeha Jul 30, 2023
0701bfc
assume the compiler server can create multiple versions when compilin…
Jakobeha Jul 30, 2023
5ca5942
try sending every function over including the baseline (bisect what d…
Jakobeha Jul 30, 2023
fe5b51f
relax assertions
Jakobeha Jul 30, 2023
90b11b3
revert to transmitting the entire DispatchTable between client and se…
Jakobeha Jul 30, 2023
7d41d4d
we can't print interned objects which are still being constructed
Jakobeha Jul 30, 2023
09b7bc0
bisect what makes the interp.cpp assertion fail...
Jakobeha Jul 30, 2023
1609ef3
bugfix?
Jakobeha Jul 31, 2023
d6ae5ac
improve pretty graph (more interactivity, better printing)
Jakobeha Jul 31, 2023
f053ae4
don't hashRoot "child" rir objects (DispatchTable contents, Function …
Jakobeha Jul 31, 2023
319919b
add ability to hide nodes which are only connected by record_call
Jakobeha Aug 2, 2023
79b770c
add ability to hide nodes which are only connected by unknown extra p…
Jakobeha Aug 2, 2023
1fb3d51
revert that (saving in case we want it)
Jakobeha Aug 2, 2023
6a43b24
don't serialize record_call extra pool entries
Jakobeha Aug 2, 2023
d32d042
misc: fix errors in clang-format migrating to python 3 so that it act…
Jakobeha Aug 3, 2023
4eee387
fix errors in gcc-13
Jakobeha Aug 3, 2023
9f9a49d
we can't skip serializing recorded calls
Jakobeha Aug 4, 2023
70c68f9
fix UUID printing to be fixed-width
Jakobeha Aug 4, 2023
cd1db8e
don't HASH record_call instructions
Jakobeha Aug 5, 2023
ea8d2a0
make Code/Function/DispatchTable not serialize children differently a…
Jakobeha Aug 5, 2023
118ffc6
hash entries which are recorded calls
Jakobeha Aug 5, 2023
67c4373
stop resetting call feedback
Jakobeha Aug 5, 2023
3e1e61b
do skip hashing recorded calls
Jakobeha Aug 5, 2023
8b7fa95
make LLVM names a global
Jakobeha Aug 5, 2023
23be0c3
rename DEBUG_SERIALIZE_LLVM to SERIALIZE_LLVM to make more clear, but…
Jakobeha Aug 5, 2023
6cafd6c
don't have to copy lambda because getOrInsertGlobal is called instant…
Jakobeha Aug 5, 2023
e0c3320
patch metadatas by replacing value names instead of the values themse…
Jakobeha Aug 6, 2023
57b9921
serialize/deserialize names correctly
Jakobeha Aug 6, 2023
3113253
add expandDotsTrigger to list of serialized globals, and make sure na…
Jakobeha Aug 6, 2023
4a048e2
hash tags in AST
Jakobeha Aug 6, 2023
1eab3c5
fix serializing names (all in hex now)
Jakobeha Aug 6, 2023
4dbc023
don't log stuff on compiler client anymore, except for startup, kill,…
Jakobeha Aug 6, 2023
08af594
fix handling metadata for functions which get optimized away (don't p…
Jakobeha Aug 7, 2023
9fc02c7
don't run certain tests (which check for properties in compiled closu…
Jakobeha Aug 7, 2023
f0eb7f7
"builtin" metadata can also contain SPECIALSXPs
Jakobeha Aug 7, 2023
8a2f416
add assertion because we're having DeoptReason Code objects without o…
Jakobeha Aug 7, 2023
c58ddff
environment variable to log interning even on client, and disable by …
Jakobeha Aug 7, 2023
b9b2932
try setting code's function earlier in case its deopt branch runs bef…
Jakobeha Aug 7, 2023
912aa53
do set function later but mark code as deserialized, so we can check …
Jakobeha Aug 7, 2023
5136bca
ok, looks like we're somehow deopting from LLVM before we finish dese…
Jakobeha Aug 7, 2023
8e26dd9
fix crashes due to suspected bug with hashes, by re-interning before …
Jakobeha Aug 8, 2023
66e910b
add assertion
Jakobeha Aug 9, 2023
63b4600
add flag to disable garbage collector
Jakobeha Aug 9, 2023
784ef77
add assertions to Function.h
Jakobeha Aug 9, 2023
00c1928
Instead of the client sending the entire closure, it sends only the A…
Jakobeha Aug 10, 2023
685bb07
@WIP new serializer API
Jakobeha Aug 10, 2023
3dd9cb2
@WIP new serializer API (draft complete)
Jakobeha Aug 11, 2023
bce9142
expose R_bcEncode and ALTREP_UNSERIALIZE_EX
Jakobeha Aug 12, 2023
0110929
@WIP bugfixing...
Jakobeha Aug 12, 2023
a1d094e
@WIP add option to disable interpreter, to confirm that we're interpr…
Jakobeha Aug 12, 2023
0196511
serialize function stats (invocation count etc)
Jakobeha Aug 12, 2023
fa68dcb
@WIP debugging...
Jakobeha Aug 12, 2023
e1f0df8
@WIP debugging...
Jakobeha Aug 12, 2023
4e8b8f5
@WIP serialize attr and tag at different points depending if we self-…
Jakobeha Aug 13, 2023
24414c5
@WIP debugging
Jakobeha Aug 13, 2023
d6b8734
merge globals from SerialRepr and other serialization, and add missin…
Jakobeha Aug 13, 2023
1fd0db1
make INTERPRETER_IS_ACTIVE a soft requirement (warn instead of error)
Jakobeha Aug 13, 2023
803f2a4
add ability for server to retrieve from client, although unlike clien…
Jakobeha Jul 23, 2023
d098b81
@WIP refactor deserializeSrc and deserializeFeedback into using deser…
Jakobeha Aug 14, 2023
b25a354
@WIP combine deserializing src and feedback for now, and remove unuse…
Jakobeha Aug 14, 2023
4c58ac4
@WIP debugging
Jakobeha Aug 14, 2023
83125b4
fix gcc UB
Jakobeha Aug 14, 2023
1c61941
add refetch command to git fetch and rebuild
Jakobeha Aug 14, 2023
a86b8b3
@WIP bugfixes
Jakobeha Aug 14, 2023
e889895
update refetch and add rerun
Jakobeha Aug 14, 2023
5081407
fix bugs so that all compiler client/server tests run
Jakobeha Aug 14, 2023
ec5b0c9
@WIP further debugging...
Jakobeha Aug 15, 2023
a606dac
don't give compiler server or client locked environments
Jakobeha Aug 15, 2023
60b3ab5
add documentation
Jakobeha Aug 15, 2023
cc14337
try also unlocking namespace environments when namespaces are deseria…
Jakobeha Aug 15, 2023
38ef91f
revert unlocking namespaces
Jakobeha Aug 15, 2023
45059fe
fix unused error code I didn't notice
Jakobeha Aug 15, 2023
1c9c897
hashing shouldn't be influenced by isChild
Jakobeha Aug 15, 2023
db7eb87
implement new hashRoot and getConnected which use the common serializ…
Jakobeha Aug 15, 2023
eb6e6fa
remove old R serialization, R serialization now uses the new serializ…
Jakobeha Aug 15, 2023
825d1d7
various fixes, passing RIR_SERIALIZE_CHAOS and compiler client/server…
Jakobeha Aug 15, 2023
27d8c59
use the old hash and getConnected for now, since they are probably fa…
Jakobeha Aug 16, 2023
0794880
fix cppcheck issues and make ByteBuffer const when we only read from it
Jakobeha Aug 16, 2023
87265b7
fix grid-Ex regression
Jakobeha Aug 16, 2023
cf17789
trying to fix weird function stats serialization/deserialization issu…
Jakobeha Aug 16, 2023
385328f
No longer print serialization differences by default (seem to be mess…
Jakobeha Aug 16, 2023
2fbe7cd
document serialization padding for sanity check
Jakobeha Aug 16, 2023
f09811c
try to fix sanitize xxhash linking error - xxhash should be built as …
Jakobeha Aug 16, 2023
43936ad
don't log UUIDPool warnings by default
Jakobeha Aug 19, 2023
b7fac01
revert OSR attempted "fix" to see if it fixes deoptless
Jakobeha Aug 19, 2023
1a778d1
try -fPIC to fix sanitizer issue
Jakobeha Aug 19, 2023
21b921a
try to fix compiler client/server gitlab test's exit code
Jakobeha Aug 19, 2023
230f8da
try to fix sanitizer build (move -fPIC to cmake)
Jakobeha Aug 19, 2023
a940f57
revert -fPIC
Jakobeha Aug 20, 2023
3143113
add regression with LLVM_SERIALIZE
Jakobeha Aug 20, 2023
a1ec584
fixed test-sanitize build
Jakobeha Aug 20, 2023
e23984a
add hash support for closure
Jakobeha Aug 21, 2023
a075c52
store LLVM symbols in SEXPs instead of `malloc` blocks, so LeakSaniti…
Jakobeha Aug 21, 2023
71ba623
do less LLVM serialize tests because it's clear they will otherwise t…
Jakobeha Aug 21, 2023
57e074e
can't run a test on gitlab because it takes too long, so I'll run it …
Jakobeha Aug 21, 2023
f0f6bc7
update test runner (add back test which took too long, but also add n…
Jakobeha Aug 21, 2023
f87cd97
use check instead of check-devel to avoid timeout
Jakobeha Aug 22, 2023
0a56f64
remove test which is probably redundant (revert unless timeout on las…
Jakobeha Aug 22, 2023
e0045c6
add parameters to log compiler client behavior
Jakobeha Aug 22, 2023
6d4745e
add more log messages
Jakobeha Aug 22, 2023
3b65649
add detailed logs
Jakobeha Aug 22, 2023
041ae9b
will never be able to skip deserializing compiled closure, because it…
Jakobeha Aug 22, 2023
988e678
indent nested response/request logs
Jakobeha Aug 22, 2023
826692c
fix rebase so it builds
Jakobeha Aug 28, 2023
4e59920
add TypeFeedback rir object to new methods
Jakobeha Aug 28, 2023
944ef8f
rebase onto extract-recording again
Jakobeha Aug 28, 2023
d169004
fix hashAst bug which existed forever but didn't cause any issues, ma…
Jakobeha Aug 28, 2023
fde3d96
@WIP bugfixes
Jakobeha Aug 29, 2023
576098a
print TypeFeedback in Function
Jakobeha Aug 29, 2023
2b1e7a4
do rename FeedbackOrigin to FeedbackPosition
Jakobeha Aug 29, 2023
e34353c
try not sending source to see what the issue is, where the feedback e…
Jakobeha Aug 29, 2023
8d04f44
debugging
Jakobeha Aug 29, 2023
1d83bce
debugging
Jakobeha Aug 30, 2023
9064fc3
debugging
Jakobeha Aug 30, 2023
696447e
debugging
Jakobeha Aug 30, 2023
479c896
debugging
Jakobeha Aug 30, 2023
e1eb0a2
suppress "Feedback should never be hashed" assertions for now
Jakobeha Aug 30, 2023
ca10d5f
set deserialized function DispatchTable, but don't serialize function…
Jakobeha Aug 30, 2023
c198ab1
Compiler server logging is also enable/disabled by environment. Renam…
Jakobeha Aug 30, 2023
498ec3b
update option to serialize source + feedback instead of the full clie…
Jakobeha Oct 6, 2023
5fd2669
for some unknown reason invocationCount, invoked, and execTime are di…
Jakobeha Oct 6, 2023
68eb7d9
refactor measuring and add ability to measure # of compiled closures …
Jakobeha Oct 6, 2023
f9bf2ac
revert not checking for differences in invocationCount / deoptCount /…
Jakobeha Oct 6, 2023
2742656
serialize function stats which are checked by rir2pir
Jakobeha Oct 7, 2023
6cb3cd0
send only source, feedback, and recorded call hashes from client to s…
Jakobeha Oct 7, 2023
bb8a1a0
@WIP draft sending server extra pool stubs and not using hashes
Jakobeha Oct 10, 2023
7946df8
@WIP also don't read and write hashes in LLVM bitcode (maybe this wil…
Jakobeha Oct 10, 2023
72fa383
@WIP debugging
Jakobeha Oct 10, 2023
7c2eb67
@WIP debugging
Jakobeha Oct 10, 2023
fa49278
@WIP pool entries are only associated with their respective code object
Jakobeha Oct 10, 2023
19b75bf
improve detailed logs
Jakobeha Oct 10, 2023
b9de430
add test to compiler-client and fix source_all_tests so that it actua…
Jakobeha Oct 10, 2023
591b23a
pretty print graph of compiled closures from env
Jakobeha Oct 10, 2023
cc97c64
@WIP
Jakobeha Oct 17, 2023
b9bb923
@WIP PIR_CLIENT_INTERN
Jakobeha Oct 20, 2023
368211d
iostream must be included with `std::cout << const char *`, did I rem…
Jakobeha Oct 20, 2023
dbe06ff
identify extra pool stubs' owner via the decompiled source code
Jakobeha Oct 20, 2023
e736c20
fix source_all_tests error reporting
Jakobeha Oct 20, 2023
fc94480
make code disassembly print LLVM module
Jakobeha Oct 20, 2023
767e052
trace serialize and deserialize
Jakobeha Oct 22, 2023
3d17bf1
improve tracing and flags
Jakobeha Oct 23, 2023
2826046
serialize default arguments pool entries as stubs too
Jakobeha Oct 23, 2023
176612d
fix serial tracing so that we don't trace reads/writes which we didn'…
Jakobeha Oct 23, 2023
1a5282e
include SEXP size in serial tracing
Jakobeha Oct 23, 2023
1389f0e
@WIP replace environments serialized across compiler peers with stubs
Jakobeha Oct 23, 2023
8b3dbfc
fix source_all_tests again...
Jakobeha Oct 23, 2023
3ae4492
fix tracing serialization and deserialization
Jakobeha Oct 24, 2023
3f1cced
add minimum size option to serial tracing
Jakobeha Oct 24, 2023
0184213
try setting closure env stub when it isn't package, namespace, or global
Jakobeha Oct 24, 2023
2ccc60e
nope, that doesn't work
Jakobeha Oct 24, 2023
f923714
print length of sexp when tracing
Jakobeha Oct 24, 2023
2e478a0
better stub (proxy) environment
Jakobeha Oct 30, 2023
6f61dc8
pass serial options in serialization/deserialization within LLVM modu…
Jakobeha Oct 30, 2023
85edde3
@WIP bugfixes
Jakobeha Oct 30, 2023
1ba6d05
serialize modules with CodeNative flags
Jakobeha Oct 30, 2023
3cc1be2
make `SerialModule` a rir object so that it gets serialized as a ref …
Jakobeha Oct 30, 2023
2cd3978
fix recursive serialization issue when using old R serialization
Jakobeha Oct 30, 2023
6906ac0
fix cppcheck issues?
Jakobeha Oct 30, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .gdbinit
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,25 @@ define ds
dumpsxp $arg0 1
end

define refetch
shell /usr/bin/git fetch mine && /usr/bin/git reset --hard mine/$(/usr/bin/git rev-parse --abbrev-ref HEAD) && /usr/bin/ninja
python gdb.execute("file " + gdb.current_progspace().filename)
directory
run
end

define rerun
python gdb.execute("file " + gdb.current_progspace().filename)
directory
run
end

define ninja
shell ninja
python gdb.execute("file " + gdb.current_progspace().filename)
directory
end

# source .pirpp.py


Expand Down
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -36,4 +36,7 @@ benchmarks/
*.DS_Store
external/*
!external/custom-r
.history
!external/xxHash
.history
.cache
compile_commands.json
121 changes: 118 additions & 3 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ tests_fullverify:
- make -j6
- bin/tests

# Test particular features, like deoptimization and serialization
# Test particular features, like deoptimization
test_features_1:
image: registry.gitlab.com/rirvm/rir_mirror:$CI_COMMIT_SHA
variables:
Expand Down Expand Up @@ -265,6 +265,73 @@ test_features_3:
when: on_failure
expire_in: 1 week

# Test serialization (no LLVM bitcode)
test_serialize_chaos:
image: registry.gitlab.com/rirvm/rir_mirror:$CI_COMMIT_SHA
variables:
GIT_STRATEGY: none
PIR_LLVM_OPT_LEVEL: 0
stage: Run tests
needs:
- rir_container
except:
- schedules
script:
- /opt/rir/container/install-test-deps.sh
- cd /opt/rir/build/release
- RIR_SERIALIZE_CHAOS=5 FAST_TESTS=1 bin/tests
- PIR_WARMUP=2 RIR_SERIALIZE_CHAOS=50 bin/gnur-make-tests check || $SAVE_LOGS
- ../../tools/check-gnur-make-tests-error
- RIR_SERIALIZE_CHAOS=10 bin/tests
artifacts:
paths:
- logs
when: on_failure
expire_in: 1 week

# Test LLVM bitcode serialization
test_serialize_llvm:
image: registry.gitlab.com/rirvm/rir_mirror:$CI_COMMIT_SHA
variables:
GIT_STRATEGY: none
PIR_LLVM_OPT_LEVEL: 0
stage: Run tests
needs:
- rir_container
except:
- schedules
script:
- /opt/rir/container/install-test-deps.sh
- cd /opt/rir/build/release
- PIR_DEBUG_SERIALIZE_LLVM=1 bin/gnur-make-tests check || $SAVE_LOGS
- ../../tools/check-gnur-make-tests-error
artifacts:
paths:
- logs
when: on_failure
expire_in: 1 week

# Test regular and LLVM bitcode serialization
test_serialize_both:
image: registry.gitlab.com/rirvm/rir_mirror:$CI_COMMIT_SHA
variables:
GIT_STRATEGY: none
PIR_LLVM_OPT_LEVEL: 0
stage: Run tests
needs:
- rir_container
except:
- schedules
script:
- /opt/rir/container/install-test-deps.sh
- cd /opt/rir/build/release
- PIR_DEBUG_SERIALIZE_LLVM=1 FAST_TESTS=1 RIR_SERIALIZE_CHAOS=5 bin/tests
- PIR_DEBUG_SERIALIZE_LLVM=1 RIR_SERIALIZE_CHAOS=10 bin/tests
artifacts:
paths:
- logs
when: on_failure
expire_in: 1 week

# Run ubsan and gc torture
test_gctorture_1:
Expand Down Expand Up @@ -334,25 +401,73 @@ test_big_inline:
test_sanitize:
image: registry.gitlab.com/rirvm/rir_mirror:$CI_COMMIT_SHA
variables:
CLANG_DIR: /opt/clang+llvm-16.0.0-x86_64-linux-gnu-ubuntu-18.04
GIT_STRATEGY: none
stage: Run tests
needs:
- rir_container
except:
- schedules
script:
# TODO: Store clang-16 and zlib on prl
# Install clang-16
- apt install libtinfo5
- wget https://github.com/llvm/llvm-project/releases/download/llvmorg-16.0.0/clang+llvm-16.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz -O $CLANG_DIR.tar.xz
- tar -xf $CLANG_DIR.tar.xz -C /opt
# Manually add zlib (required when building on clang-16)
- wget https://github.com/madler/zlib/releases/download/v1.3/zlib-1.3.tar.xz -O /opt/rir/external/zlib-1.3.tar.xz
- tar -xf /opt/rir/external/zlib-1.3.tar.xz -C /opt/rir/external
- cd /opt/rir/external/zlib-1.3
- ./configure --prefix=/opt/rir/external/zlib
- make -j6
- make install
# Rest
- curl 10.200.14.25:8080/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz > /opt/rir/external/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz
- mkdir /opt/rir/build/sanitize
- cd /opt/rir/build/sanitize
- /opt/rir/tools/fetch-llvm.sh
- CC=$(ls ../../external/clang*/bin/clang) CXX=$(ls ../../external/clang*/bin/clang) cmake -DCMAKE_BUILD_TYPE=sanitize ../..
- cmake -DCMAKE_BUILD_TYPE=sanitize
-DCMAKE_C_COMPILER=$CLANG_DIR/bin/clang
-DCMAKE_CXX_COMPILER=$CLANG_DIR/bin/clang++
-DZLIB_LIBRARY=/opt/rir/external/zlib/lib/libz.so
-DZLIB_INCLUDE_DIR=/opt/rir/external/zlib/include
../..
- make -j6
# R_LD_PRELOAD is a feature of the test-runner. To repro this without the testrunner use LD_PRELOAD instead.
# intercept_tls_get_addr=0 helps with leak sanitizer crashes
- ASAN_OPTIONS="intercept_tls_get_addr=0" LSAN_OPTIONS="symbolize=1" ASAN_SYMBOLIZER_PATH=$(ls /opt/rir/external/clang*/bin/llvm-symbolizer) R_LD_PRELOAD=$(ls /opt/rir/external/clang*/lib/clang/12.0.0/lib/linux/libclang_rt.asan-x86_64.so) bin/tests
- ASAN_OPTIONS="intercept_tls_get_addr=0"
LSAN_OPTIONS="symbolize=1"
ASAN_SYMBOLIZER_PATH=$CLANG_DIR/bin/llvm-symbolizer
R_LD_PRELOAD=$CLANG_DIR/lib/clang/16/lib/x86_64-unknown-linux-gnu/libclang_rt.asan.so
bin/tests
# sometimes leak sanitizer segfaults
retry: 2

# Test the compiler server and client (on localhost)
test_compiler_server_client:
image: registry.gitlab.com/rirvm/rir_mirror:$CI_COMMIT_SHA
variables:
GIT_STRATEGY: none
PIR_LLVM_OPT_LEVEL: 0
stage: Run tests
needs:
- rir_container
except:
- schedules
script:
- mkdir $CI_PROJECT_DIR/results && mkdir $CI_PROJECT_DIR/results/debug && mkdir $CI_PROJECT_DIR/results/release
- curl 10.200.14.25:8080/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz > /opt/rir/external/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz
- /opt/rir/tools/fetch-llvm.sh
- mkdir /opt/rir/build/debug && cd /opt/rir/build/debug && cmake -DCMAKE_BUILD_TYPE=Debug ../.. && make -j6
- PIR_CLIENT_SKIP_DISCREPANCY_CHECK=1 ./bin/test-compiler-client-and-server; cp /tmp/test-compiler-server-actual.out /tmp/test-compiler-client-actual.out $CI_PROJECT_DIR/results/debug/
- cd /opt/rir/build/release
- PIR_CLIENT_SKIP_DISCREPANCY_CHECK=1 ./bin/test-compiler-client-and-server; cp /tmp/test-compiler-server-actual.out /tmp/test-compiler-client-actual.out $CI_PROJECT_DIR/results/release/
artifacts:
paths:
- results
when: always
expire_in: 1 week

# Test the benchmarks container before deploying
test_benchmarks:
image: registry.gitlab.com/rirvm/rir_mirror/benchmark:$CI_COMMIT_SHA
Expand Down
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,6 @@
path = external/custom-r
url = https://github.com/reactorlabs/gnur.git
ignore = untracked
[submodule "external/xxHash"]
path = external/xxHash
url = https://github.com/Cyan4973/xxHash.git
91 changes: 47 additions & 44 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ set(R_LIBRARY_TREE ${CMAKE_SOURCE_DIR}/packages)
set(R_ROOT_DIR ${R_HOME})
set(R_INCLUDE_DIR ${R_HOME}/include)
set(LLVM_DIR ${CMAKE_SOURCE_DIR}/external/llvm-12)
set(ZEROMQ_DIR ${CMAKE_SOURCE_DIR}/external/zeromq)
set(XXHASH_DIR ${CMAKE_SOURCE_DIR}/external/xxHash)

set(R_COMMAND ${R_HOME}/bin/R)

Expand All @@ -22,29 +24,28 @@ endif ()
include_directories(SYSTEM ${LLVM_INCLUDE_DIRS})
add_definitions(${LLVM_DEFINITIONS})

# use GCC 9 on macOS. Otherwise we use clang
if(APPLE)
option(MACOS_USE_GCC_9 "Use GCC 9 on macOS." FALSE)
endif(APPLE)
include_directories(${ZEROMQ_DIR}/include)
# Won't work on Linux if we try to link statically:
# /usr/bin/ld: ../external/zeromq/lib/libzmq.a(libzmq_la-zmq.o): relocation R_X86_64_PC32 against symbol
# `_ZSt7nothrow@@GLIBCXX_3.4' can not be used when making a shared object; recompile with -fPIC
if (${APPLE})
link_libraries(${ZEROMQ_DIR}/lib/libzmq.dylib)
else ()
link_libraries(${ZEROMQ_DIR}/lib/libzmq.so)
endif ()

if(${MACOS_USE_GCC_9})
set(CMAKE_C_COMPILER /usr/local/bin/gcc-9 CACHE PATH "" FORCE)
set(CMAKE_CXX_COMPILER /usr/local/bin/g++-9 CACHE PATH "" FORCE)
endif()
set(XXHASH_BUILD_XXHSUM OFF)
set(BUILD_SHARED_LIBS ON)
add_subdirectory(${XXHASH_DIR}/cmake_unofficial/ ${XXHASH_DIR}/build/ EXCLUDE_FROM_ALL)

add_definitions(-g)
set(CMAKE_CXX_FLAGS_RELEASE "-O2 -Werror -DSWITCH_TO_NAMED=1")
set(CMAKE_CXX_FLAGS_RELEASENOASSERT "${CMAKE_CXX_FLAGS_RELEASE} -DNDEBUG")
set(CMAKE_CXX_FLAGS_FULLVERIFIER "${CMAKE_CXX_FLAGS_RELEASE} -DFULLVERIFIER")
set(CMAKE_CXX_FLAGS_RELEASESLOWASSERT "${CMAKE_CXX_FLAGS_RELEASE} -DENABLE_SLOWASSERT")
set(CMAKE_CXX_FLAGS_DEBUG "-O0 -DSWITCH_TO_NAMED=1 -DENABLE_SLOWASSERT")
set(CMAKE_CXX_FLAGS_DEBUG "-O0 -DSWITCH_TO_NAMED=1 -DENABLE_SLOWASSERT -DDEBUG_DISASSEMBLY")
set(CMAKE_CXX_FLAGS_DEBUGOPT "-Og -DSWITCH_TO_NAMED=1 -DENABLE_SLOWASSERT")
# with macOS GCC 9 we need to explicitly use libc++, since llvm does. See https://libcxx.llvm.org/docs/UsingLibcxx.html#using-libc-with-gcc
if(${MACOS_USE_GCC_9})
set(CMAKE_CXX_FLAGS_LIBCXX "-nostdinc++ -nodefaultlibs -lc++ -lc++abi -lm -lc -lgcc_s.1 -lgcc")
else()
set(CMAKE_CXX_FLAGS_LIBCXX "")
endif()
set(CMAKE_CXX_FLAGS_LIBCXX "")
set(CMAKE_CXX_FLAGS "${LLVM_CXX_FLAGS} ${CMAKE_CXX_FLAGS_LIBCXX} -Wall -Wuninitialized -Wundef -Winit-self -Wcast-align -Woverloaded-virtual -Wmissing-include-dirs -Wstrict-overflow=3 -std=c++14 -fno-rtti -fno-exceptions -Wimplicit-fallthrough -Wno-deprecated-declarations")
set(CMAKE_C_FLAGS_RELEASE "-O2 -DSWITCH_TO_NAMED=1")
set(CMAKE_C_FLAGS_RELEASENOASSERT "${CMAKE_C_FLAGS_RELEASE} -DNDEBUG")
Expand All @@ -54,8 +55,16 @@ set(CMAKE_C_FLAGS_DEBUG "-O0 -DSWITCH_TO_NAMED=1 -DENABLE_SLOWASSERT")
set(CMAKE_C_FLAGS_DEBUGOPT "-Og -DSWITCH_TO_NAMED=1 -DENABLE_SLOWASSERT")
set(CMAKE_C_FLAGS "-std=gnu99")

set(CMAKE_CXX_FLAGS_SANITIZE "${CMAKE_CXX_FLAGS_RELEASE} -g2 -fno-omit-frame-pointer -fsanitize=undefined -fsanitize=address -fsanitize=leak -fno-sanitize=alignment -shared-libasan -fvisibility=default")
set(CMAKE_C_FLAGS_SANITIZE "${CMAKE_C_FLAGS_RELEASE} -g2 -fno-omit-frame-pointer -fsanitize=undefined -fsanitize=address -fsanitize=leak -fno-sanitize=alignment -shared-libasan -fvisibility=default")
if (${APPLE})
set(SANITIZE_FLAGS "-g2 -fno-omit-frame-pointer -fsanitize=undefined -fsanitize=address -fno-sanitize=alignment -shared-libasan -fvisibility=default")
else()
set(SANITIZE_FLAGS "-g2 -fno-omit-frame-pointer -fsanitize=undefined -fsanitize=address -fsanitize=leak -fno-sanitize=alignment -shared-libasan -fvisibility=default")
endif()
set(CMAKE_CXX_FLAGS_SANITIZE "${CMAKE_CXX_FLAGS_RELEASE} ${SANITIZE_FLAGS}")
set(CMAKE_C_FLAGS_SANITIZE "${CMAKE_C_FLAGS_RELEASE} ${SANITIZE_FLAGS}")
set(CMAKE_CXX_FLAGS_DEBUG_SANITIZE "${CMAKE_CXX_FLAGS_DEBUG} ${SANITIZE_FLAGS}")
set(CMAKE_C_FLAGS_DEBUG_SANITIZE "${CMAKE_C_FLAGS_DEBUG} ${SANITIZE_FLAGS}")


MARK_AS_ADVANCED(
CMAKE_CXX_FLAGS_SANITIZE
Expand Down Expand Up @@ -96,12 +105,15 @@ endif()

# Create proxy scripts for the scripts in /tools
file(MAKE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/.bin_create")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/tests" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/tests \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/R" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/R \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/Rscript" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/Rscript \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/Rgnu" "#!/bin/sh\n${CMAKE_SOURCE_DIR}/external/custom-r/bin/R \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/gnur-make" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/gnur-make \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/gnur-make-tests" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/gnur-make-tests \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/tests" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/tests \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/test-compiler-client-and-server" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/test-compiler-client-and-server \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/test-compiler-client-only" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/test-compiler-client-only \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/test-compiler-server-only" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/test-compiler-server-only \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/R" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/R \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/Rscript" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/Rscript \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/Rgnu" "#!/bin/sh\n${CMAKE_SOURCE_DIR}/external/custom-r/bin/R \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/gnur-make" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/gnur-make \"$@\"")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/gnur-make-tests" "#!/bin/sh\nRIR_BUILD=\"${CMAKE_CURRENT_BINARY_DIR}\" ${CMAKE_SOURCE_DIR}/tools/gnur-make-tests \"$@\"")

file(GLOB BIN_IN "${CMAKE_CURRENT_BINARY_DIR}/.bin_create/*")
file(INSTALL ${BIN_IN} DESTINATION "${CMAKE_CURRENT_BINARY_DIR}/bin" FILE_PERMISSIONS OWNER_EXECUTE OWNER_READ GROUP_READ GROUP_EXECUTE)
Expand All @@ -118,36 +130,20 @@ endif(${NO_LOCAL_CONFIG})

include_directories(${R_INCLUDE_DIR})
include_directories(${CMAKE_SOURCE_DIR}/rir/src)
# again we need to explicitly use libc++
if(${MACOS_USE_GCC_9})
include_directories(/Library/Developer/CommandLineTools/usr/include/c++/v1)
endif(${MACOS_USE_GCC_9})
message(STATUS "Using R from ${R_HOME}")

add_custom_target(setup-build-dir
COMMAND ${CMAKE_SOURCE_DIR}/tools/setup-build-dir ${CMAKE_SOURCE_DIR} ${R_HOME}
)

if(${MACOS_USE_GCC_9})
add_custom_target(dependencies
COMMAND ${CMAKE_SOURCE_DIR}/tools/build-gnur.sh --macos_gcc9
COMMAND ${CMAKE_SOURCE_DIR}/tools/fetch-llvm.sh --macos_gcc9
)
else()
add_custom_target(dependencies
COMMAND ${CMAKE_SOURCE_DIR}/tools/build-gnur.sh
COMMAND ${CMAKE_SOURCE_DIR}/tools/fetch-llvm.sh
)
endif()

add_custom_target(default-gnur
DEPENDS dependencies
COMMAND ${CMAKE_SOURCE_DIR}/tools/build-gnur.sh custom-r
add_custom_target(dependencies
COMMAND ${CMAKE_SOURCE_DIR}/tools/build-gnur.sh
COMMAND ${CMAKE_SOURCE_DIR}/tools/build-zeromq.sh
COMMAND ${CMAKE_SOURCE_DIR}/tools/fetch-llvm.sh
)

add_custom_target(setup
DEPENDS dependencies
DEPENDS default-gnur
)

add_custom_target(tests
Expand Down Expand Up @@ -175,10 +171,17 @@ if (DEFINED LLVM_PACKAGE_VERSION)
target_link_libraries(${PROJECT_NAME} ${LLVM_LIBS})
endif(DEFINED LLVM_PACKAGE_VERSION)

target_link_libraries(${PROJECT_NAME} xxHash::xxhash)

if(APPLE)
set_target_properties(${PROJECT_NAME} PROPERTIES LINK_FLAGS "-L${R_HOME}/lib")
target_link_libraries(${PROJECT_NAME} R)
# to resolve build error from
# https://www.gnu.org/software/gettext/FAQ.html#integrating_undefined
target_link_libraries(${PROJECT_NAME} -lintl)
# target_link_libraries(${PROJECT_NAME} -lintl)
include_directories(${LLVM_DIR}/include)
target_link_directories(${PROJECT_NAME} INTERFACE ${LLVM_DIR}/lib)
# Note: May need to update the version here
include_directories(/opt/homebrew/Cellar/gettext/0.21.1/include)
target_link_libraries(${PROJECT_NAME} /opt/homebrew/Cellar/gettext/0.21.1/lib/libintl.dylib)
endif(APPLE)
5 changes: 4 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,16 @@ ENV LANG en_US.UTF-8
RUN echo $CI_COMMIT_SHA > /opt/rir_version && \
apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get upgrade -y -qq && \
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq curl git gcc gfortran g++ libreadline-dev libx11-dev libxt-dev zlib1g-dev libbz2-dev liblzma-dev libpcre3-dev libcurl4-openssl-dev libcairo2-dev make libreadline8 libncurses-dev xz-utils cmake tcl-dev tk-dev locales rsync && \
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq curl git gcc gfortran g++ libreadline-dev libx11-dev libxt-dev zlib1g-dev libbz2-dev liblzma-dev libpcre3-dev libcurl4-openssl-dev libcairo2-dev make libreadline8 libncurses-dev xz-utils cmake tcl-dev tk-dev locales rsync wget && \
locale-gen en_US.UTF-8 && update-locale LANG=en_US.UTF-8 && \
cd /opt/rir && \
tools/build-gnur.sh && \
rm -rf external/custom-r/cache_recommended.tar .git && \
find external -type f -name '*.o' -exec rm -f {} \; && \
apt-get clean
RUN cd /opt/rir && \
USE_NINJA=0 tools/build-zeromq.sh && \
rm -rf external/zeromq-* external/cppzmq-*
RUN mkdir -p /opt/rir/build/release && \
cd /opt/rir && \
(curl 10.200.14.25:8080/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz > external/clang+llvm-12.0.0-x86_64-linux-gnu-ubuntu-20.04.tar.xz || true) && \
Expand Down
Loading