Seminar on Adversarial Reinforcement Learning
Saarland University — Winter Semester 2021
The course will provide an overview of recent research in Adversarial Reinforcement Learning (RL). Research papers covered in the course will showcase the landscape of attacks on RL agents and the optimal attack strategies, which is crucial for understanding security threats against the deployed systems. In particular, the research papers will cover optimal attack strategies for test-time, backdoor, and training-time (environment poisoning) attacks on RL agents. After this course, the participants will gain a better perspective of important problems for developing robust and secure algorithms in sequential decision-making settings.

Organizers

Timeline and updates

  • Until 15 Oct 2021: Register for the seminar course at https://seminars.cs.uni-saarland.de.
  • Until 25 Nov 2021: After you have been allocated a slot in the seminar, you then need to register for the seminar course examination at UdS. You should check with UdS when the examination registration starts; you need to register for the seminar course examination by 25 Nov 2021.
  • 31 Oct 2021: Paper assignments for reading and writing reports is available to students. Each student is assigned a total of 8 papers for which they will be writing reports. The list of papers is provided below and each student has the same set of papers.
  • 2 Nov 2020: We have a new mailing list that includes all the organizers/tutors. To reach out to us, please send an email to adversarialrl-w21-tutors@mpi-sws.org (instead of contacting individuals).
  • 30 Nov 2021: Reports for the three papers on test-time attacks (#1, #2, #3) are due.
  • 15 Dec 2021: Reports for the two papers on backdoor attacks (#4, #5) are due.
  • 15 Jan 2022: Reports for the three papers on training-time attacks (#6, #7, #8) are due.
  • 20 Jan 2022: Paper assignments for presentations will be sent to students. One paper will be assigned to each student that they will be presenting.
  • 15 Feb 2022: Presentation slides are due.
  • Between 16 Feb to 5 Mar 2022: Final presentations will take place where each student will present their assigned paper. The exact dates will be finalized in discussion with enrolled students.

Course structure

The course consists of two main components: (i) Reading research papers and writing reports; (ii) Presentations. There will be no weekly classes, and the course will operate in the format of a block seminar. If you have any questions, you can contact us by sending an email to adversarialrl-w21-tutors@mpi-sws.org. If needed, the tutors will arrange specific meeting times during the semester — further information will be communicated to students via emails as we move along in the semester.

Reading research papers and writing reports

  • Each student is assigned a total of 8 research papers. These papers cover test-time, backdoor, and training-time attacks with 2-3 papers per topic. The list of papers is provided below and each student has the same set of papers.
  • For each of the assigned papers, you will have to write a short (up to 2 pages) report. The timeline for report submissions is listed above.
  • Each report should be submitted as a PDF file via email to adversarialrl-w21-tutors@mpi-sws.org. You should name your PDF files as lastname_#.pdf (i.e., lastname_1.pdf, lastname_2.pdf, lastname_3.pdf, lastname_4.pdf, lastname_5.pdf, lastname_6.pdf, lastname_7.pdf, and lastname_8.pdf).
  • Reports should be written in latex using NeurIPS style files
  • Structure the report as an extended review, e.g.,
    • Summarize the paper.
    • Write down the main strengths of the paper.
    • Write down the main weaknesses of the paper.
    • Write down ways in which this paper could be improved.
    • Write down ideas in which this paper could be extended.
  • These 8 reports will correspond to 60% of the final score.

Presentations

  • Each student will be assigned a paper for presentation. This paper will be selected from the list of 8 papers.
  • You will have to prepare a presentation of 25 mins. You will have the possibility to get feedback on your slides before the final submission.
  • At the end of the semester, you will give a final presentation. We will block about 6 hours of time for the presentations. The exact dates will be finalized in discussion with enrolled students.
  • The slides and presentation will correspond to 40% of the final score.

List of research papers

Test-time attacks

  1. Tactics of Adversarial Attack on Deep Reinforcement Learning Agents
    by Y. Lin, Z. Hong, Y. Liao, M. Shih, M. Liu, and M. Sun, at IJCAI 2017.
  2. Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learning
    by J. Sun, T. Zhang, X. Xie, L. Ma, Y. Zheng, K. Chen, and Y. Liu, at AAAI 2020.
  3. Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
    by H. Zhang, H. Chen, C. Xiao, B. Li, M. Liu, D. Boning, and C. Hsieh, at NeurIPS 2020.

Backdoor attacks

  1. TrojDRL: Evaluation of Backdoor Attacks on Deep Reinforcement Learning
    by P. Kiourti, K. Wardega, S. Jha, and W. Li, at DAC 2020.
  2. Temporal Watermarks for Deep Reinforcement Learning Models
    by K. Chen, S. Guo, T. Zhang, S. Li, and Y. Liu, at AAMAS 2021.

Training-time attacks

  1. Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
    by A. Rakhsha, G. Radanovic, R. Devidze, X. Zhu, and A. Singla, at ICML 2020.
  2. Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics
    by Y. Sun, D. Huo, and F. Huang, at ICLR 2021.
  3. Defense Against Reward Poisoning Attacks in Reinforcement Learning
    by K. Banihashem, A. Singla, and G. Radanovic, at arXiv preprint 2021.


Imprint / Data Protection