The build-it, break-it, fix-it project takes place over three phases. It is a team project; teams should have about 4 people.
Phase 1: 4 weeks. Build-it. Teams build software according to a problem specification that is announced on the day the project begins. This software must pass a set of correctness tests that we will provide. It is scored based on its performance, and on how many optional features it implements.
Phase 2: 4 weeks. Break-it. Teams are given access to the source code of build-it submissions and they attempt to find correctness and security defects (where the definition of the latter is problem specific). They submit test cases that provide evidence of these defects, and these are vetted by our submission system. Teams receive points for each defect they find, and more points for security-relevant defects. The team in whose submission the defect is found loses points.
Phase 3: 2 weeks. Fix-it. Teams are given the test cases demonstrating defects found in their submission, and may fix those defects. If fixing a defect causes multiple test cases to pass (as is likely) then the fixing team earn points back for the duplicate tests (and the breaking teams which found the defect lose some points).
All projects must compile and build on this VirtualBox-compatible Ubuntu Linux virtual machine. The username for the VM is ubuntu, and the password is ubuntu. You can download Virtual Box for free. You can also get a free VMware player and convert the VM to that, if you wish.
If you find that our Ubuntu image does not have a software package installed that you would like, let us know by emailing pcr@tamu.edu and we will update it. No package requests will be accepted after 8am on Monday 16 March.
As packages are accumulated by requests, we will distribute a script to allow you to easily install them to update your local VM.
A defect is incorrect behavior by a program.
A bug is a defect caused by a problem in the code.
A flaw is a defect caused by a problem in the design.
A vulnerability is an exploitable defect in a program.
A crash is an unexpected program termination (a violation of availability).
A security violation is a violation of confidentiality or integrity.
We will use "bug" to mean a vulnerability which does not violate security goals (i.e. is just a correctness problem).
Required tests (250 pts total). All teams' submissions must pass a set of correctness and performance tests. If you correctly pass all tests, you get 250 points.
Optional tests (25 pts each + performance bonuses). These tests are not required, but they are a great way to earn extra points.
There are two types of tests:
Correctness tests -- These test the correctness of specification features. Each is worth 25 points.
Performance tests -- These are larger tests that do more substantial work. For each test case, you get up to 50 extra points depending on how efficient your submission is. The amount of points you get is scaled linearly based on the fastest and slowest submissions. For example, if your submission takes 15 seconds to run, the fastest submission takes 10 seconds to run, and the slowest submission takes 20 seconds to run, you would receive 25 points. Tying submissions get an equal number of points, and timeouts receive no points.
Teams receive 25 points for identifying a bug, crash, or security violation in the oracle implementation. Points will only be awarded to the first team that reports the defect. To report a defect in the oracle implementation, email us with a description of the defect, a test case demonstrating the defect, and your team name.
For every unique vulnerability found in your submission that leads to a correctness bug, you will lose 25 points.
For every unique vulnerability found in your submission that leads to a crash, you will lose 50 points.
For every unique vulnerability found in your submission that leads to a security violation, you will lose 100 points.
(For what "unique" means, see the Fix-it scoring below.)
The total number of points received for any bug found is 25.
The total number of points received for any crash found is 50.
The total number of points received for any security violation found is 100.
The points from a given bug are split evenly among all the teams who find it.
Example -- Each bug is worth 25 points. If there are 5 teams who find a given bug, then they each get 5 points; if you are the only team to find a given bug, you get the full 25 points.
Some of the reported vulnerabilities found by several other teams might correspond to the same vulnerability. If the building team can prove that multiple vulnerabilities are the same, they will gain points back such that they are only penalized for the bug vulnerability once.
To show that multiple reported vulnerabilities are actually the "same," submit a single, small code patch that removes the vulnerability. Judges will use these fixes to determine which vulnerabilities are "basically the same".
Be careful to only fix one unique vulnerability at a time. You will be penalized if you fix more than one unique vulnerability at a time.
Do not attack the project's infrastructure. Malicious attacks may result in an honor code violation.
Teams must do their own work and not share solutions with others.
Instructional staff reserve the right to use their discretion in making judgements.
Do not obfuscate the source code in your submissions. The penalty for source obfuscation is a 0 on the build-it phase.
Teams have a per-test time limit of 10 minutes and time limit of 30 minutes for all tests to complete.
Do not change your build-it submission between the build-it and fix-it phases. Any changes made will be considered as part of your first fix, which may end up disqualifying it.
Teams are limited to 5 vulnerability ("break") submissions against a single team.
Suggested strategy: find many breaks against each team, then pick the best 5. Since points for breaks are split between all teams that submit the vulnerability, "best" can be highest valued, most complex, difficult to find, etc.
Only fix a single vulnerability at a time.