IOI2001 Questionnaire for Competitors Summary of further comments written on 119 returned questionnaires [Remarks in square brackets are mine, Tom Verhoeff] ======= Suggestions for improving the Linux competition environment ----------- Lynx should be installed, so you do not have to use X-system, because switching between X and console is not good for the screens. Please install Joe next time. I would enjoy a cleaner Linux system. WindowMaker was popular in '95, and KDE/GNOME are really good. Especially KDevelop with C++ STL reference would be useful. And a higher screen resolution, because X11 fonts (DDD) are BIG. Midnight Commander, Joe, help system (man pages, STL docs, info pages, info in RHIDE). Help only available via RHIDE; should also be available with the 'man' and 'info' commands. The mouse was flaky; it would sometimes stop responding under X-system after switching from the console. Some X-system editors specialized for programming would be appreciated. With code highlighting and indentation. Others could get some CDs with Linux, so it would become more popular. [And we could do an introductory Linux workshop for interested delegation leaders and competitors.] MC [= Midnight Commander] I want Midnight Commander. Add 'mc' (Midnight Commander) tool: very useful for quick viewing and editing of files, and copying input data (cf. task DOUBLE). Add manual!!! Remove bugs: mouse did not work in text mode and, after switching between X-system and tty several times. There was no Joe editor. There was no mouse on console. There were problems using ALT and CTRL keys in RHIDE under X-system. Install MC, GPM, manual pages for C functions. Make switching between console and X-system work properly. I dearly missed having GPM running under Linux (console mouse support). It would be nice to have CTRL/ALT-bound hotkeys work with RHIDE under X-system as well. For competitors that like the text-only terminal and command line, command-line submission or submission by a text-only webbrowser (lynx) could be available. I think that GNOME is better than WindowMaker. It worked rather well, but using GNOME instead of WindowMaker would be better. It would be better to include a Task Bar. Include graphical editors (e.g. grim), more choices of debuggers, and an on-line STL manual. Add STL help. Allow RHIDE to access help files. Install more window managers, as I at least am not familiar with WindowMaker. I would find KDE, fvwm, or twm more convenient to use. RHIDE should be properly configured for FreePascal. Make FTE editor available. RHIDE should be preconfigured for FreePascal. It does work, but just needs the right config; there is info on freepascal.org). Debugging with FreePascal IDE should be possible. Correct keyboard settings. Fix: FP does not find linker. FP and mouse in X. Add KDE or GNOME. Linux operating system is very good, I think (and much better than Windows), but is has no good developing tool (RHIDE has to many bugs). The IDEs should work better. RHIDE not able to use mouse in console. Numerical keypad did not work. Not able to change video mode. Suggestions for improving the Windows competition environment ----------- Remove useless shortcuts, such as Outlook, on the desktop and and Quickstart bar. Remove it. Do not use it. Since standard FreePascal IDE hardly managess to debug programs correctly and easily, at least an alternative external debugger should be available for Windows. FreePascal IDE is very unstable. FreePascal IDE debugging did not work correctly. Debugging under Windows was quite difficult (Pascal), especially when the given library could not be used in TP7 and I had to use FP debugger, which is full of bugs. FP help did not work. FP debugger is too slow. The Windows environment was very good. The FreePascal IDE crashed many times. FAR would be better than command.com. It was almost impossible for me to run the FreePascal IDE for Windows, because it is too slow in responding to mouse or keyboard. Navigator tool was missing (Windows Commander or FAR would be fine. File associations for .in and .out files could be set to Notepad. FreePascal (Windows) did not work properly. Tasks were not solvable in Turbo Pascal. FAR, WinCommander. I had problems with FreePascal IDE debugger. Suggestions for improving the Grading System and Web Services ----------- No need to submit before backup [?] Shorter name in Restore, or provide links to backup files Auto-backup all submissions You should get the possibility to submit a game which does not run according to the rules, because it might be that the game plays according to the rules in some other examples. The web service is not very usable. You should get the possibility - to run each time you want to test/submit an example - to save the program on a NFS device. You should get the results by regular e-mail too (like in the practice competition before July 14th) When you try to submit a program which is not accepted, the old one should not be deleted. Submitting (and printing, testing, ...) by console would be better in my opinion. I do not like the idea of submitting solutions with webbrowser. I know that that is easier to implement (and it is portable between Windows and Linux), but I personall like console programs. And when it has to be on the web, I missed these features: - option to see program I have submitted - test a solution without old one being deleted I do not like the idea of deleting an old solution before a new one has been accepted successfully. The grading took too long. Why? I would like some 'force' option when submitting a solution which does not work on simple input file. I think Backup/Restore would be easier for you and us, when simply setting up an FTP server. Do not delete submissions until new submission passes the test. Although I knew the behavior of the system, and kept my own backups, I still panicked on this a few times. Some way to make test runs on a competitor's computer and get a core file... I did not use Print and Test. I do not like the idea of the web feature: I could not submit outputs of DOUBLE task (server not responding). It should be improved. After the competition we could not save input, output, source files: we could only see it in web browser. During the competition there was some problem with the Restore feature. It would be better to submit programs from command line (not using web browser). Backup/Restore were difficult to use, because restored programs were part of an HTML file. Also my submitted programs and test inputs were available only as part of an HTML file and uncomfortable ot use them. Our programs in HTML file were not shown correctly, because text between < and > was treated as an HTML tag. It would be better if submitted programs, inputs, correct outputs and our outputs were available as plain text files. Make it possible to Submit/Print/Backup/Restore/Test using the command line. Restoring backup files and watching sources and test data should show only the file, without menus and headers. Detailed reports should include all the details, not only correctness of output, like in MOBILES. The retrieval of backup files could text-only. It is annoying that you have to remove the web page header and footer from the saved file. Or, it could return other data type (content type), like application/octet-stream, then the web browser would ask if the user wants to open or save the file. In my humble opinion, there is no need to compile/test the submitted source code on each submission. The competitors could test the programs on their own computers, and, if needed, they could use the Test web service. When a successful Test is done, there could be an option to "submit this tested source code", after the Test. The Test web service could show the output of the program, while testing it. The label "Program File" on the main page could be changed to "Solution File (source or output)", or something like that. I did not understand WHY the previously submitted program is deleted, when testing the program. The servers will be less overloaded, if we can test the program without losing the previously submitted program and without compiling-on-submission. I would like more a grading system available from the command line. It would be better to use the ACM task submission system. [PC^2] Limit the compile time only during competition, not for grading. Make compile and test an OPTION in the submission system. Provide a way to test reactive tasks, e.g. the grading system could pipe two programmes to each other and report the results. Do not kill off previous submission when you click Test. I did not use Print/Backup/Restore/Test. Improve speed of uploading answers. Make list available of backups for restoring. When testing, the submitted files should not get erased. Test function should not overwrite submitted data. You could have made exectime source available. Testing should not clobber upload. Submitting broken solution should not clobber previous upload. Want access to backed up stuff afterwards. Exectime should not add 50ms to execution time. Or, if exectime is used, some mechanism should be used to compensate for it, so it does not penalize you. It would be nice to have web access to the input for tasks like DOUBLE, as well as the tools, in case local copies are accidentally deleted. It should be made clear which time (wall clock, user/user+sys) is used for timing. Server for submitting is sometimes too busy. The web submission system was very good. It would be better though if testing did not delete the submission. Overall it was not too slow. I did not use Print/Backup/Restore. The test facility did not work correctly: it did not show the output. I did not get any print. Provide tools to view submitted programs and files. Provide tools to save backup files into harddisk. Submission program may use other test data (instead of sample data) to test competitors' programs. Many problems with the network, notably poor server response and slow submission time. In addition, submitting in the last 5 to 10 minutes was a real pain. More bandwidth maybe? The web-based testing and submission was nice, because then you would not have to worry about formatting problesm, incorrect input files, etc. Old submitted files should NOT be deleted if a newer submission fails. Submitted and backup files should be available after the competition together with the results. All result data should be downloadable. Complete result data should be available: input/output, all comments and correct output. A solution to all tasks should be given to the competitors, at the same time as the results. It would be good if the previous submission would not be deleted if the new one fails or if I submit my program for test execution. More servers, so we can submit better and faster. Accept wrong solutions, even though they cannot compile. [And do what with them?] Restore is rather troublesome, having to edit all your HTML code out. Slow or server down sometimes. I would really have liked to be able to get a backup of all source code that I left on my harddrive. It would have been great if a submission automatically would have equalled a "backup", so to give me the opportunity to restore my previous submissions. If I submit a program, and the server compiles it, but it does not correspond, it should not delete my previously submitted program. If you Test a program, it should not replace or delete the submitted one. You can see which backup files you have (easily, just like the submitted files). Submitted files are backed up. Use the compiler switches which are in the overview sheet we got (Pascal: fpc -So (Turbo Pascal compatibility), or do not put it there. The second day the grading system was MUCH faster (especially compared with the last fifteen minutes on the first day). Backup submitted files. Other IDE/editors ----- XEmacs Vim Gedit (2x) MS/DOS edit (5x) notepad (18x) wordpad (3x) Debuggers used --------- RHIDE integrated debugger (25x) gdb (19x) ddd (3x) rhgdb gdb integrated into FP IDE (12x) using extra writeln statements Turbo Pascal integrated debugger (4x) Other Linux tools used ----- bash (5x) bc (13x) own shell scripts time (6x) ulimit cat grep (3x) standard Linux command-line utilities (3x) calculator date gcalc bash scripts sed man (2x) xcalc (3x) GNOME calculator Textutils info perl (to generate test cases) nn [?], objdump, and gdb's disassemble command (on IOIWARI and DOUBLE) objdump make mkfifo Other Windows tools used ----- calculator (16x) command.com fc (file compare) winmine [??] ASCII table Competition environment (general) ----------- Everyone should get the possibility to choose his keyboard, because I always swapped 'Y' and 'Z', and in one program I used variables named 'X', 'Y' and 'Z', so I had a serious bug in my program which I found only the last hour! The messages should come by a local program on everyone's computer screen, not by speakers (so you can read the messages twice when you don't understand them). Don't confuse 'fify' and 'fifteen'. You should also provide rulers. [For drawing straight lines and measuring distances?] You should not write into the rules that you provide pens, when you do not provide some! Every competitor should get an 'example solution'. [When, how?] The keyboards were very uncomfortable: keys were too flat; Home, PageDown, PageUp, Insert, Delete, End keys were lower than usual. Keyboards were uncomfortable (keys were too flat; Insert, Delete and other keys were at different place than usual). The keyboard was ok, except the block with Ins, Del, ... was shifted down one row: not good. Would have liked to access my stuff after the second competition day. Keyboards should not have the Home/End/PgUp/PgDn/Insert/Delete block in the wrong place. Every competitor should be asked about his/her preferred keyboard, and get it. Order some PC/Mac compatible hardware. [??] I need STL help. It would be quite useful to provide random-input generator with each task, i.e. you would fill in the numbers (N, M, S, ...) and the tool would generate a random input data set using these numbers. Tasks ----- DOUBLE: Nice task, but not well-written task description. If you tell that 4*s is number of valid bits and other bits are zero, then some people understand it that 4*s bits must not be zero. I was wondering where's the problem for 1.5 hours! And then I had no time to finish SCORE. [The English task description called them 'relevant' bits. If they would have been non-zero, it would have been explicitly expressed as 'non-zero' bits. But you are right, it could have been stated that the relevant bits are possibly zero. Since the task was the easiest of all, we left it for you to discover. Lesson: Do not jump to conclusions.] I do not think that games like IOIWARI and SCORE are good in an algorithm competition like this, because to find an optimal strategy there are two ways: you already know a game like that, or you must play the game. The tasks seemed a bit "unconventional" this year. In task SCORE, it was not written that it is a DAG. [You could derive that from what was given.] MOBILES: Some algorithms did not work because of slow output instructions (they would have with file output). That should have been specified. Some legal test data was impossible to solve (60000 output test file, which did not work even with a program that only had output instruction, because of time limit). Miscellaneous ------------- The question and answer period of one hour is really too short. [We need to look into this.] The practice competition was too short. There should be more computers with internet access (one computer for everybody) and computers in English, not Finnish, and in Linux, not in Windows. And you should get the possibility to program during free time (on the computers I mentioned before). It was, and is still, great time here in Tampere. It was a great competition. Thanks. In general, you did GREAT. However, generally it was good. There was not enough food during the competition. After the competition, official solutions should be available. Official solutions (programs) should be available after the competition. I think that you should add Java to the list of allowed programming languages. The competition organizer should prepare more sandwiches during the competition. Since the competition is so long, you get too hungry. Better choice of breakfast. Thanks for the mouse. (2x) Mina rakastan sinua! [Actually 'Mina' with a-umlaut (ä) ??] Mina en rakasta sinua!!!!! [??] Computers with internet and programming tools should be available for competitors in sufficient number and all the time (in English). Question time is too short. Practice competition should be longer. I wish to subscribe to the IOI mailing list. Can you return the code after the competition?