Skip to content

Commit 437b647

Browse files
Merge branch 'develop-test-libraries'. Close #475.
**Description** As part of the requirements for Class D (NPR7150.2C), we need unit tests for all of copilot. A natural next step is to introduce the testing infrastructure for `copilot-libraries`. **Type** - Management: conformance with new requirement. **Additional context** None. **Requester** - Ivan Perez **Method to check presence of bug** The following Dockerfile checks that copilot-libraries's tests can be executed, in which case it prints the message Success: ``` FROM ubuntu:focal RUN apt-get update RUN apt-get install --yes libz-dev RUN apt-get install --yes git RUN apt-get install --yes wget RUN mkdir -p $HOME/.ghcup/bin RUN wget https://downloads.haskell.org/~ghcup/0.1.19.2/x86_64-linux-ghcup-0.1.19.2 -O $HOME/.ghcup/bin/ghcup RUN chmod a+x $HOME/.ghcup/bin/ghcup ENV PATH=$PATH:/root/.ghcup/bin/ ENV PATH=$PATH:/root/.cabal/bin/ RUN apt-get install --yes curl RUN apt-get install --yes gcc make libgmp3-dev g++ SHELL ["/bin/bash", "-c"] RUN ghcup install ghc 9.4.7 RUN ghcup install cabal 3.8 RUN ghcup set ghc 9.4.7 RUN cabal update RUN apt-get install --yes z3 CMD git clone $REPO && \ cd $NAME && \ git checkout $COMMIT && \ cd copilot-libraries && \ cabal test copilot-libraries:unit-tests && \ echo Success ``` Command (substitute variables based on new path after merge): ``` $ docker run -e "REPO=https://github.com/Copilot-Language/copilot" -e "NAME=copilot" -e "COMMIT=<HASH>" -it copilot-verify-475 ``` **Expected result** The library `copilot-libraries` includes tests for some of the existing modules. Introducing (some classes of) errors in the implementation of `copilot-libraries` makes the tests detect those errors. Running the docker file above prints the message success, indicating that copilot-libraries now declares a unit-tests component. **Solution implemented** Add tests for `copilot-libraries` that test a basic property in the module Copilot.Libraries.PTLTL. The property is tested using both copilot-theorem and copilot-interpreter, showing how both interfaces can be used. **Further notes** Due to the magnitude of this change, we decide to simplify the issue by merely introducing the infrastructure. We leave it to a future issue to include a more comprehensive list of properties that covers all modules.
2 parents fbfbbd3 + 15ca577 commit 437b647

File tree

7 files changed

+685
-4
lines changed

7 files changed

+685
-4
lines changed

.travis.yml

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ script:
3333
#
3434
# Only libraries with tests are listed below or the v2-test command fails.
3535
#
36-
# Testing copilot-theorem requires z3. See above conditional installation,
37-
# and keep GHC version numbers in both places in sync.
38-
- if [ "${GHCVER}" == "8.10.4" ]; then cabal v2-test -j1 copilot-core copilot-language copilot-interpreter copilot-c99 copilot-theorem; fi
36+
# Testing copilot-theorem and copilot-libraries requires z3. See above
37+
# conditional installation, and keep GHC version numbers in both places in
38+
# sync.
39+
- if [ "${GHCVER}" == "8.10.4" ]; then cabal v2-test -j1 copilot-core copilot-language copilot-interpreter copilot-c99 copilot-theorem copilot-libraries; fi

copilot-libraries/CHANGELOG

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,6 @@
1+
2023-12-24
2+
* Introduce testing infrastructure for Copilot.Library. (#475)
3+
14
2023-11-07
25
* Version bump (3.17). (#466)
36

copilot-libraries/copilot-libraries.cabal

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,3 +57,34 @@ library
5757

5858
ghc-options:
5959
-Wall
60+
61+
test-suite unit-tests
62+
type:
63+
exitcode-stdio-1.0
64+
65+
main-is:
66+
Main.hs
67+
68+
other-modules:
69+
Test.Copilot.Library.PTLTL
70+
Test.Extra
71+
72+
build-depends:
73+
base
74+
, QuickCheck
75+
, test-framework
76+
, test-framework-quickcheck2
77+
78+
, copilot-interpreter
79+
, copilot-language
80+
, copilot-libraries
81+
, copilot-theorem
82+
83+
hs-source-dirs:
84+
tests
85+
86+
default-language:
87+
Haskell2010
88+
89+
ghc-options:
90+
-Wall

copilot-libraries/tests/Main.hs

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
-- | Test copilot-libraries.
2+
module Main where
3+
4+
-- External imports
5+
import Test.Framework (Test, defaultMain)
6+
7+
-- Internal imports
8+
import qualified Test.Copilot.Library.PTLTL
9+
10+
-- | Run all unit tests on copilot-libraries.
11+
main :: IO ()
12+
main = defaultMain tests
13+
14+
-- | All unit tests in copilot-libraries.
15+
tests :: [Test.Framework.Test]
16+
tests =
17+
[ Test.Copilot.Library.PTLTL.tests
18+
]
Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
-- | Test copilot-libraries:Copilot.Library.PTLTL
2+
module Test.Copilot.Library.PTLTL
3+
(tests)
4+
where
5+
6+
-- External imports
7+
import Test.Framework (Test, testGroup)
8+
import Test.Framework.Providers.QuickCheck2 (testProperty)
9+
import Test.QuickCheck (Gen, Property)
10+
11+
-- External imports: Copilot
12+
import Copilot.Language (extern)
13+
import qualified Copilot.Language.Operators.Boolean as Copilot
14+
import Copilot.Language.Stream (Stream)
15+
import Copilot.Theorem.What4 (SatResult (..))
16+
17+
-- Internal imports: auxiliary functions
18+
import Test.Extra (arbitraryBoolExpr, testWithInterpreter, testWithTheorem)
19+
20+
-- Internal imports: Modules being tested
21+
import Copilot.Library.PTLTL (eventuallyPrev, previous)
22+
23+
-- * Constants
24+
25+
-- | Unit tests for copilot-libraries:Copilot.Library.PTLTL.
26+
tests :: Test.Framework.Test
27+
tests =
28+
testGroup "Copilot.Library.PTLTL"
29+
[ testProperty "previous x ==> eventuallyPrev x (theorem)"
30+
testProvePreviousEventually
31+
, testProperty "previous x ==> eventuallyPrev x (interpreter)"
32+
testCheckPreviousEventually
33+
]
34+
35+
-- * Individual tests
36+
37+
-- | Test that Z3 is able to prove the following expression valid:
38+
-- @
39+
-- previous x ==> eventuallyPrev x
40+
-- @
41+
testProvePreviousEventually :: Property
42+
testProvePreviousEventually = testWithTheorem pair
43+
where
44+
pair :: Gen (Stream Bool, SatResult)
45+
pair = pure (stream, expectation)
46+
47+
stream :: Stream Bool
48+
stream =
49+
previous boolStream Copilot.==> eventuallyPrev boolStream
50+
where
51+
boolStream = extern "x" Nothing
52+
53+
expectation :: SatResult
54+
expectation = Valid
55+
56+
-- | Test that the following stream is always true:
57+
-- @
58+
-- previous x ==> eventuallyPrev x
59+
-- @
60+
testCheckPreviousEventually :: Property
61+
testCheckPreviousEventually = testWithInterpreter pair
62+
where
63+
pair :: Gen (Stream Bool, [Bool])
64+
pair = do
65+
-- We discard the expectation from the expression; the temporal formula
66+
-- holds at all times regardless.
67+
boolStream <- fst <$> arbitraryBoolExpr
68+
let prop = previous boolStream Copilot.==> eventuallyPrev boolStream
69+
return (prop, expectation)
70+
71+
expectation :: [Bool]
72+
expectation = repeat True

0 commit comments

Comments
 (0)