Editing VAX (section)

== Instruction set ==
The VAX instruction set was designed to be powerful and [[Orthogonal instruction set|orthogonal]].<ref name="Levy">{{Cite book|url=https://books.google.com/books?id=3LijBQAAQBAJ|title=Computer Programming and Architecture: The Vax|last1=Levy|first1=Henry|last2=Eckhouse|first2=Richard|date=June 28, 2014|publisher=Digital Press|isbn=9781483299372|language=en}}</ref> When it was introduced, many programs were written in assembly language, so having a "programmer-friendly" instruction set was important.<ref>{{cite web
|title=Another Approach to Instruction Set Architecture—VAX
|url=http://www.weblearn.hs-bremen.de/risse/RST/docs/COD2e/WebExtns.ion/webext3.pdf
|quote=... instruction set architectures, we chose the VAX as programmer-friendly instruction set, an asset
|access-date=October 3, 2018
|archive-date=June 10, 2017
|archive-url=https://web.archive.org/web/20170610060129/http://www.weblearn.hs-bremen.de/risse/RST/docs/COD2e/WebExtns.ion/webext3.pdf
|url-status=dead
}}</ref><ref>{{cite web |title=VAX |quote=Esp. noted for its large, assembler-programmer-friendly instruction set --- an asset that|url=http://www.lysator.liu.se/hackdict/split/vax.html}}</ref> In time, as more programs were written in [[high-level programming language]]s, the instruction set became less visible, and the only ones much concerned about it were compiler writers.

One unusual aspect of the VAX instruction set is the presence of register masks<ref>{{cite web |url=http://www.openvms.compaq.com/doc/73final/4515/4515pro_020.html#index_x_790 |title=VAX MACRO and Instruction Set Reference Manual |at=9.2.5 Procedure Call Instructions |date=April 2001 |work=OpenVMS documentation |archive-url=https://web.archive.org/web/20020330111145/http://www.openvms.compaq.com/doc/73final/4515/4515pro_020.html#index_x_790 |archive-date=March 30, 2002 |url-status=dead}}</ref> at the start of each subprogram. These are arbitrary bit patterns that specify, when control is passed to the subprogram, which registers are to be preserved. On most architectures, it is up to the compiler to produce instructions to save out the needed data, typically using the [[call stack]] for temporary storage. On the VAX, with 16 registers, this might require 16 instructions to save the data and another 16 to restore it. Using the mask, a single 16-bit value performs the same operations internally in hardware, saving time and memory.<ref name="Levy"/>

Since register masks are a form of data embedded within the executable code, they can make linear parsing of the machine code difficult. This can complicate optimization techniques that are applied on machine code.<ref name="MCO">{{cite thesis |last=Goss |first=Clinton F. |date=August 2013 |title=Machine Code Optimization: Improving Executable Object Code |type=PhD |orig-date=First published June 1986 |volume=Computer Science Department Technical Report No. 246 |publisher=Courant Institute, New York University |url=http://www.ClintGoss.com/mco/Goss_1986_MachineCodeOptimization.pdf |access-date=August 22, 2013 |arxiv=1308.4815 |bibcode=2013arXiv1308.4815G}}
*{{cite thesis |degree=PhD |author=Clinton F. Goss |date=2013 |orig-date=1986 |title=Machine Code Optimization – Improving Executable Object Code | publisher=Courant Institute, New York University |url=http://www.ClintGoss.com/mco/}}</ref>