Skip to content

Commit 8331f58

Browse files
jacobmsftjacobmsft
authored
Merge pull request #8 from microsoft/dev/jacobmsft/docs
Add more functionality to the scripts, add new generic script, add to…
2 parents ddbe643 + c6b3f73 commit 8331f58

File tree

6 files changed

+267
-42
lines changed

6 files changed

+267
-42
lines changed

‎README.md‎

Lines changed: 34 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -21,13 +21,13 @@ We keep updating the docker image periodically and uploading it to the Microsoft
2121

2222
You can pull the image by running the command:
2323
```
24-
$ docker pull mcr.microsoft.com/codeql/codeql-container
24+
$ docker pull mcr.microsoft.com/cstsectools/codeql-container
2525
```
2626

2727
If you want to analyze a particular source directory with codeql, run the container as:
2828

2929
```
30-
$ docker run --rm --name codeql-container mcr.microsoft.com/codeql/codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS=<query run...>
30+
$ docker run --rm --name codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS=<query run...> mcr.microsoft.com/cstsectools/codeql-container
3131
```
3232

3333
where `/dir/to/analyze` contains the source files that have to be analyzed, and `/dir/for/results` is where the result output
@@ -59,9 +59,9 @@ For example, if you want to analyze a python project source code placed in `/dir
5959
to analyze and get a SARIF result file, you will have to run:
6060

6161
```
62-
$ docker run --rm --name codeql-container mcr.microsoft.com/codeql/codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS="database create --language=python /opt/src/source_db"
63-
$ docker run --rm --name codeql-container mcr.microsoft.com/codeql/codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS=" database upgrade /opt/src/source_db"
64-
$ docker run --rm --name codeql-container mcr.microsoft.com/codeql/codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS="database analyze --format=sarifv2 --output=/opt/results/issues.sarif /opt/src/source_db"
62+
$ docker run --rm --name codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS="database create --language=python /opt/src/source_db /opt/output/source_db" mcr.microsoft.com/cstsectools/codeql-container
63+
$ docker run --rm --name codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS=" database upgrade /opt/src/source_db" mcr.microsoft.com/cstsectools/codeql-container
64+
$ docker run --rm --name codeql-container -v /dir/to/analyze:/opt/src -v /dir/for/results:/opt/results -e CODEQL_CLI_ARGS="database analyze --format=sarifv2 --output=/opt/results/issues.sarif /opt/src/source_db" mcr.microsoft.com/cstsectools/codeql-container
6565
```
6666

6767
For more information on CodeQL and QL packs, please visit https://www.github.com/github/codeql.
@@ -76,6 +76,35 @@ cd codeql-container
7676
docker build . -f Dockerfile -t codeql-container
7777
```
7878

79+
# Convenience Scripts
80+
Analyzing a source directory takes multiple invocations of the container, as mentioned above. To help with that, we've built some scripts for convenience, which does these invocations for you.
81+
These scripts are in the ```scripts``` folder, under their respective platforms (unix or windows).
82+
83+
84+
##### analyze_security.sh
85+
scripts/unix/analyze_security.sh (or scripts/windows/analyze_security.bat for windows) runs the Security and Quality QL pack suite on your project. This is how you would run it:
86+
87+
```
88+
scripts/unix/analyze_security.sh /path/to/analyze /path/to/results language
89+
```
90+
91+
For example for the python project can be analyzed thus:
92+
93+
```
94+
/scripts/unix/analyze_security.sh /tmp/django/src /tmp/django/output python
95+
```
96+
97+
for JavaScript:
98+
```
99+
/scripts/unix/analyze_security.sh /tmp/express/src /tmp/express/output javascript
100+
```
101+
102+
##### run_qlpack.sh
103+
If you know which QL suite you would like to run on the code to be analyzed, use scripts/unix/run_qlpack.sh (or scripts/windows/run_qlpack.bat for windows).
104+
```
105+
scripts/unix/analyze_security.sh /path/to/analyze /path/to/results language qlpack
106+
```
107+
79108
# Contributing
80109

81110
This project welcomes contributions and suggestions. Most contributions require you to agree to a

‎container/libs/codeql.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ def precompile_queries(self):
8888
execute_codeql_command(f' query compile --search-path {self.CODEQL_HOME} {self.CODEQL_HOME}/codeql-repo/*/ql/src/codeql-suites/*.qls')
8989

9090
def execute_codeql_command(self, args):
91-
ret_string = check_output_wrapper(f'{self.CODEQL_HOME}/codeql/codeql {args}', shell=True).decode("utf-8")
91+
ret_string = check_output_wrapper(f'{self.CODEQL_HOME}/codeql/codeql {args}', shell=True)
9292
if ret_string is CalledProcessError:
9393
logger.error("Could not run codeql command")
9494
exit(ERROR_EXECUTING_CODEQL)

‎scripts/unix/analyze_security.sh‎

Lines changed: 62 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -2,22 +2,69 @@
22
scriptname=$(basename "$0")
33
inputfile=${1}
44
outputfile=${2}
5+
language=${3}
56

6-
if [ "$#" -ne 2 ]; then
7-
echo "Please provide the folder to analyze, and the folder to store results"
8-
echo "Usage: ${scriptname} <folder to analyze> <folder to store result>"
7+
RED="\033[31m"
8+
YELLOW="\033[33m"
9+
GREEN="\033[32m"
10+
RESET="\033[0m"
11+
12+
print_yellow() {
13+
echo -e "${YELLOW}${1}${RESET}"
14+
}
15+
16+
print_red() {
17+
echo -e "${RED}${1}${RESET}"
18+
}
19+
20+
print_green() {
21+
echo -e "${GREEN}${1}${RESET}"
22+
}
23+
24+
if [ "$#" -ne 3 ]; then
25+
print_yellow "\nPlease provide the folder to analyze, the folder to store results, and the coding language of the project."
26+
print_yellow "\nUsage: ${scriptname} <folder to analyze> <folder to store result> <language>"
27+
print_yellow "\nExample: ${scriptname} /tmp/pandas /tmp/results python"
28+
exit 1
29+
fi
30+
31+
print_yellow "Getting/Updating the codeQL container\n"
32+
docker pull mcr.microsoft.com/cstsectools/codeql-container:latest
33+
if [ $? -eq 0 ]
34+
then
35+
print_green "\nPulled the container"
36+
else
37+
print_red "\nFailed to pull container"
938
exit 1
1039
fi
1140

12-
RED=''
13-
RESET=''
14-
#docker pull sargemonkey/codeql-container
15-
#[ $? -eq 0 ] && echo "Pulled the container" || echo -e "failed to pull container";exit 1
16-
docker run --rm --name codeql-container -v "${inputfile}:/opt/src" -v "${outputfile}:/opt/results" -e CODEQL_CLI_ARGS=database\ create\ --language=python\ /opt/src/source_db csteosstools.azurecr.io/codeql/codeql-container
17-
[ $? -eq 0 ] && echo "Created the database" || echo -e "\n${RED}Failed to create the database${RESET}\n";exit 1
18-
docker run --rm --name codeql-container -v "${inputfile}:/opt/src" -v "${outputfile}:/opt/results" -e CODEQL_CLI_ARGS=database\ upgrade\ /opt/src/source_db csteosstools.azurecr.io/codeql/codeql-container
19-
[ $? -eq 0 ] && echo "Upgraded the database" || echo -e "\n${RED}failed to upgrade the database${RESET}\n";exit 2
20-
docker run --rm --name codeql-container -v ${inputfile}:/opt/src -v ${outputfile}:/opt/results -e CODEQL_CLI_ARGS=database\ analyze\ /opt/src/source_db\ --format=sarifv2\ --output=/opt/results/issues.sarif\ python-security-and-quality.qls csteosstools.azurecr.io/codeql/codeql-container
21-
[ $? -eq 0 ] && echo "Query execution successful" || echo -e "\n${RED}Query execution failed${RESET}\n"; exit 3
22-
23-
echo "The results file should be located at ${2}/issues.sarif"
41+
print_yellow "\nCreating the codeQL database. This might take some time depending on the size of the project..."
42+
docker run --rm --name codeql-container -v "${inputfile}:/opt/src" -v "${outputfile}:/opt/results" -e CODEQL_CLI_ARGS=database\ create\ --language=${3}\ /opt/results/source_db\ -s\ /opt/src mcr.microsoft.com/cstsectools/codeql-container
43+
if [ $? -eq 0 ]
44+
then
45+
print_green "\nCreated the database"
46+
else
47+
print_red "\nFailed to create the database"
48+
exit 1
49+
fi
50+
51+
docker run --rm --name codeql-container -v "${inputfile}:/opt/src" -v "${outputfile}:/opt/results" -e CODEQL_CLI_ARGS=database\ upgrade\ /opt/src/source_db mcr.microsoft.com/cstsectools/codeql-container
52+
if [ $? -eq 0 ]
53+
then
54+
print_green "\nUpgraded the database\n"
55+
else
56+
print_red "\nFailed to upgrade the database"
57+
exit 2
58+
fi
59+
60+
print_yellow "\nRunning the Quality and Security rules on the project"
61+
docker run --rm --name codeql-container -v ${inputfile}:/opt/src -v ${outputfile}:/opt/results -e CODEQL_CLI_ARGS=database\ analyze\ /opt/src/source_db\ --format=sarifv2\ --output=/opt/results/issues.sarif\ ${language}-security-and-quality.qls mcr.microsoft.com/cstsectools/codeql-container
62+
if [ $? -eq 0 ]
63+
then
64+
print_green "\nQuery execution successful"
65+
else
66+
print_red "\nQuery execution failed\n"
67+
exit 3
68+
fi
69+
70+
[ $? -eq 0 ] && print_yellow "The results are saved at ${2}/issues.sarif"

‎scripts/unix/run_ql_suite.sh‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
#!/bin/bash
2+
scriptname=$(basename "$0")
3+
inputfile=${1}
4+
outputfile=${2}
5+
language=${3}
6+
qlpack=${4}
7+
8+
RED="\033[31m"
9+
YELLOW="\033[33m"
10+
GREEN="\033[32m"
11+
RESET="\033[0m"
12+
13+
print_yellow() {
14+
echo -e "${YELLOW}${1}${RESET}"
15+
}
16+
17+
print_red() {
18+
echo -e "${RED}${1}${RESET}"
19+
}
20+
21+
print_green() {
22+
echo -e "${GREEN}${1}${RESET}"
23+
}
24+
25+
if [ "$#" -ne 4 ]; then
26+
print_yellow "\nPlease provide the folder to analyze, the folder to store results, the coding language of the project, and the QL suite to run."
27+
print_yellow "\nUsage: ${scriptname} <folder to analyze> <folder to store result> <language> <ql suite>"
28+
print_yellow "\nExample: ${scriptname} /tmp/pandas /tmp/results python security-and-quality"
29+
exit 1
30+
fi
31+
32+
33+
print_yellow "Getting/Updating the codeQL container\n"
34+
docker pull mcr.microsoft.com/cstsectools/codeql-container:latest
35+
if [ $? -eq 0 ]
36+
then
37+
print_green "\nPulled the container"
38+
else
39+
print_red "\nFailed to pull container"
40+
exit 1
41+
fi
42+
43+
print_yellow "\nCreating the codeQL database. This might take some time depending on the size of the project..."
44+
docker run --rm --name codeql-container -v "${inputfile}:/opt/src" -v "${outputfile}:/opt/results" -e CODEQL_CLI_ARGS=database\ create\ --language=${3}\ /opt/results/source_db\ -s\ /opt/src mcr.microsoft.com/cstsectools/codeql-container
45+
if [ $? -eq 0 ]
46+
then
47+
print_green "\nCreated the database"
48+
else
49+
print_red "\nFailed to create the database"
50+
exit 1
51+
fi
52+
53+
docker run --rm --name codeql-container -v "${inputfile}:/opt/src" -v "${outputfile}:/opt/results" -e CODEQL_CLI_ARGS=database\ upgrade\ /opt/src/source_db mcr.microsoft.com/cstsectools/codeql-container
54+
if [ $? -eq 0 ]
55+
then
56+
print_green "\nUpgraded the database\n"
57+
else
58+
print_red "\nFailed to upgrade the database"
59+
exit 2
60+
fi
61+
62+
print_yellow "\nRunning the Quality and Security rules on the project"
63+
docker run --rm --name codeql-container -v ${inputfile}:/opt/src -v ${outputfile}:/opt/results -e CODEQL_CLI_ARGS=database\ analyze\ /opt/src/source_db\ --format=sarifv2\ --output=/opt/results/issues.sarif\ ${language}-${qlpack}.qls mcr.microsoft.com/cstsectools/codeql-container
64+
if [ $? -eq 0 ]
65+
then
66+
print_green "\nQuery execution successful"
67+
else
68+
print_red "\nQuery execution failed\n"
69+
exit 3
70+
fi
71+
72+
[ $? -eq 0 ] && print_yellow "The results are saved at ${2}/issues.sarif"

‎scripts/windows/analyze_security.bat‎

Lines changed: 36 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,46 +1,61 @@
1+
@echo off
2+
setlocal enabledelayedexpansion
3+
14
set scriptname=%0
25
set inputfile=%1
36
set outputfile=%2
4-
5-
@echo off
6-
setlocal enabledelayedexpansion
7+
set language=%3
78

89
set argCount=0
910
for %%x in (%*) do (
1011
set /A argCount+=1
1112
set "argVec[!argCount!]=%%~x"
1213
)
1314

14-
if %argCount% LSS 2 (
15-
echo "Please provide the folder to analyze, and the folder to store results"
16-
echo "Usage: %scriptname% <folder to analyze> <folder to store result>"
15+
if %argCount% LSS 3 (
16+
call :print_yellow "Please provide the folder to analyze, the folder to store results, and the coding language of the project"
17+
call :print_yellow "Usage: %scriptname% :folder to analyze: :folder to store result: :language:"
18+
call :print_yellow "Example: %scriptname% C:\Source\pandas C:\Results python"
1719
exit /b 1
1820
)
1921

20-
rem docker pull codeql/codeql-container
21-
echo docker run --rm --name codeql-container -v "%inputfile%:/opt/src" -v "%outputfile%:/opt/results" -e CODEQL_CLI_ARGS="database create --language=python /opt/src/source_db" csteosstools.azurecr.io/codeql/codeql-container
22-
start /W /B docker run --rm --name codeql-container -v "%inputfile%:/opt/src" -v "%outputfile%:/opt/results" -e CODEQL_CLI_ARGS="database create --language=python /opt/src/source_db" csteosstools.azurecr.io/codeql/codeql-container
22+
call :print_yellow "Getting the image..."
23+
docker pull mcr.microsoft.com/cstsectools/codeql-container
24+
call :print_green "Pulled the container"
25+
26+
call :print_yellow "Creating the codeQL database. This might take some time depending on the size of the project..."
27+
start /W /B docker run --rm --name codeql-container -v "%inputfile%:/opt/src" -v "%outputfile%:/opt/results" -e CODEQL_CLI_ARGS="database create --language=%language%% /opt/src/source_db -s /opt/src" mcr.microsoft.com/cstsectools/codeql-container
2328

24-
call :print_status "Failed creating the database" , %errorlevel%
2529
if %errorlevel% GTR 0 (
26-
call :print_exit_error "Failed creating the database"
30+
call :print_red "Failed creating the database"
2731
exit /b %errorlevel%
2832
)
29-
start /W /B docker run --rm --name codeql-container -v "%inputfile%:/opt/src" -v "%outputfile%:/opt/results" -e CODEQL_CLI_ARGS="database upgrade /opt/src/source_db" csteosstools.azurecr.io/codeql/codeql-container
33+
34+
start /W /B docker run --rm --name codeql-container -v "%inputfile%:/opt/src" -v "%outputfile%:/opt/results" -e CODEQL_CLI_ARGS="database upgrade /opt/src/source_db" mcr.microsoft.com/cstsectools/codeql-container
3035
if %errorlevel% GTR 0 (
31-
call :print_exit_error "Failed upgrading the database"
36+
call :print_red "Failed upgrading the database"
3237
exit /b %errorlevel%
3338
)
34-
start /W /B docker run --rm --name codeql-container -v "%inputfile%:/opt/src" -v "%outputfile%:/opt/results" -e CODEQL_CLI_ARGS="database analyze /opt/src/source_db --format=sarifv2 --output=/opt/results/issues.sarif python-security-and-quality.qls" csteosstools.azurecr.io/codeql/codeql-container
39+
40+
call :print_yellow "Running the Quality and Security rules on the project"
41+
start /W /B docker run --rm --name codeql-container -v "%inputfile%:/opt/src" -v "%outputfile%:/opt/results" -e CODEQL_CLI_ARGS="database analyze /opt/src/source_db --format=sarifv2 --output=/opt/results/issues.sarif %language%-security-and-quality.qls" mcr.microsoft.com/cstsectools/codeql-container
3542
if %errorlevel% GTR 0 (
36-
call :print_exit_error "Failed to run the query on the database"
43+
call :print_red "Failed to run the query on the database"
3744
exit /b %errorlevel%
3845
)
39-
echo "The results file should be located at %2\issues.sarif"
4046

47+
if %errorlevel% EQU 0 (
48+
call :print_yellow "The results file are saved at at %2\issues.sarif"
49+
)
50+
51+
:print_yellow
52+
echo %~1
53+
exit /b 0
54+
55+
:print_red
56+
echo %~1
57+
exit /b 0
4158

42-
:print_exit_error
43-
echo.
44-
echo %~1
45-
echo.
46-
echo Exiting...
59+
:print_green
60+
echo %~1
61+
exit /b 0

‎scripts/windows/run_ql_suite.bat‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
@echo off
2+
setlocal enabledelayedexpansion
3+
4+
set scriptname=%0
5+
set inputfile=%1
6+
set outputfile=%2
7+
set language=%3
8+
set qlpack=%4
9+
10+
set argCount=0
11+
for %%x in (%*) do (
12+
set /A argCount+=1
13+
set "argVec[!argCount!]=%%~x"
14+
)
15+
16+
if %argCount% LSS 4 (
17+
call :print_yellow "Please provide the folder to analyze, the folder to store results, coding language of the project, and the QL suite"
18+
call :print_yellow "Usage: %scriptname% :folder to analyze: :folder to store result: :language: :QL suite:"
19+
call :print_yellow "Example: %scriptname% C:\Source\pandas C:\Results python security-and-quality"
20+
exit /b 1
21+
)
22+
23+
call :print_yellow "Getting the image..."
24+
docker pull mcr.microsoft.com/cstsectools/codeql-container
25+
call :print_green "Pulled the container"
26+
27+
call :print_yellow "Creating the codeQL database. This might take some time depending on the size of the project..."
28+
start /W /B docker run --rm --name codeql-container -v "%inputfile%:/opt/src" -v "%outputfile%:/opt/results" -e CODEQL_CLI_ARGS="database create --language=%language%% /opt/src/source_db -s /opt/src" mcr.microsoft.com/cstsectools/codeql-container
29+
30+
if %errorlevel% GTR 0 (
31+
call :print_red "Failed creating the database"
32+
exit /b %errorlevel%
33+
)
34+
35+
start /W /B docker run --rm --name codeql-container -v "%inputfile%:/opt/src" -v "%outputfile%:/opt/results" -e CODEQL_CLI_ARGS="database upgrade /opt/src/source_db" mcr.microsoft.com/cstsectools/codeql-container
36+
if %errorlevel% GTR 0 (
37+
call :print_red "Failed upgrading the database"
38+
exit /b %errorlevel%
39+
)
40+
41+
call :print_yellow "Running the Quality and Security rules on the project"
42+
start /W /B docker run --rm --name codeql-container -v "%inputfile%:/opt/src" -v "%outputfile%:/opt/results" -e CODEQL_CLI_ARGS="database analyze /opt/src/source_db --format=sarifv2 --output=/opt/results/issues.sarif %language%-%qlpack%.qls" mcr.microsoft.com/cstsectools/codeql-container
43+
if %errorlevel% GTR 0 (
44+
call :print_red "Failed to run the query on the database"
45+
exit /b %errorlevel%
46+
)
47+
48+
if %errorlevel% EQU 0 (
49+
call :print_yellow "The results file are saved at at %2\issues.sarif"
50+
)
51+
52+
:print_yellow
53+
echo %~1
54+
exit /b 0
55+
56+
:print_red
57+
echo %~1
58+
exit /b 0
59+
60+
:print_green
61+
echo %~1
62+
exit /b 0

0 commit comments

Comments
 (0)