|
From: Thomas R. <tr...@st...> - 2010-12-14 09:40:50
|
Hi all
[domjudge cc'd in the hope that you're interested in such
developments]
First of all thanks for a great tool. In some years of using
valgrind, I only found a small bug in it, whereas it found lots of
bugs in my own code ;-)
Some back story (skip it if you're in a hurry):
I am part of a team of assistants giving a course called "Algorithms
Lab" at ETH. It works vaguely similar to the ACM contests: the
students solve a programming problem and hand it in at a "judge"[1].
The judge (actually a little cluster of 3*4 cores) automatically
compiles and executes the problem on a set of predefined (secret)
test cases, validates the output and saves the result in a database.
The students can then, with only slight delay, see the result.
The problem is that, to enforce use of efficient algorithms and also
to protect the judge against DoS, we impose hard time limits. As I
am sure you are well aware, times() and friends are not a very
precise way of measuring time, thus the same program may hit the
timelimit only some of the time. CPU stepping has made it really
hard to do any timings; programs run slower on the cluster when it
runs hot than when it only tests a single program. Finally, all of
this would of course completely break down on the day we add some
new hardware.
So at some point this year I figured, maybe valgrind could help with
this, after all it already does instruction counts. And it seems to
work pretty well[2]:
git://honeybunny.inf.ethz.ch/valgrind master
I started from nulgrind and added pieces of memcheck and other tools
until happy. It now outputs a score of 1 per every 1e7 IR register
accesses (Put[I]/Get[I]), which puts it in the same order of magnitude
as running the same programs unsupervised on my laptop. The huge
benefit of course is that this score is deterministic and the same
across hardware (if not compilers/platforms). The downside of course
is that I get a slowdown factor of about 8, but then again even
best-of-5 timings would incur a slowdown of 5 and still not give
deterministic results.
The slight downside is that in this setting valgrind is (abused as) a
security-relevant piece of software. I tried to plug all holes that I
could think of:
* Syscall whitelisting is done within valgrind, notably preventing the
client from escaping valgrind through execve(), and from closing or
duping over the FD used for valgrind messages
* Client requests are banned (immediate exit when seeing them in the IR)
* Memory is tracked far enough to be sure that the client cannot write
to valgrind's memory
As soon as you can stop screaming "valgrind -- security -- CRAZY", I
would of course be interested to hear if there are any other known
holes. I still plan on keeping the "other" syscall whitelist, to have
double security against leaking the testcases through socket() etc,
but any issue with valgrind would at least allow the student to
circumvent the time limits, which is bad enough.
Barring any major issues, I am hoping to roll this out by next
September in time for the next instalment of the course.
Regards,
Thomas
[1] It runs a fork of DOMjudge.
[2] Posting git links seems the norm here? But I haven't found an
up-to-date "authoritative" git-svn clone, so I took
git://gitorious.org/valgrind/mainlinemirror.git
and then fetched the rest myself. I can also post patches if you
like.
--
Thomas Rast
trast@{inf,student}.ethz.ch
|
|
From: Julian S. <js...@ac...> - 2011-01-03 13:51:30
|
Interesting use case! > * Syscall whitelisting is done within valgrind, notably preventing the > client from escaping valgrind through execve(), and from closing or > duping over the FD used for valgrind messages > > * Client requests are banned (immediate exit when seeing them in the IR) > > * Memory is tracked far enough to be sure that the client cannot write > to valgrind's memory > > As soon as you can stop screaming "valgrind -- security -- CRAZY", I > would of course be interested to hear if there are any other known > holes. I still plan on keeping the "other" syscall whitelist, to have > double security against leaking the testcases through socket() etc, > but any issue with valgrind would at least allow the student to > circumvent the time limits, which is bad enough. I'm not sure if it could be used to circumvent time limits, but .. you might want to stop the client looking at /proc/self/* or even within /proc. Using (eg) /proc/self/maps it is easy to figure out that you're not running on the real hardware. We already have to fake /proc/self/cmdline (iirc) in order to make OpenOffice work at all. J |