Data Collection & Creation
A wide variety of public agency, private, and open-sourced data was utilized to create a robust dataset including information from 2000-2021 across hundreds of rules and thousands of comment letters/meetings.
1.1 Rules & Comment Letters
The text of all Proposed and Final Rules in addition to the associated comment letters and meeting memorandums between 2000 and 2021 was collected. The text was then parsed to create a novel dataset of textual information about each textual element (e.g. word count).
Every Proposed and Final Rule, also by requirement, are published on the Federal Register and SEC website. Through the use of Federal Register numbers (FRNumbers) a Proposed Rule can be associated with a Final Rule (if adopted) following the notice-and-comment period. Within the text of every Proposed and Final Rule there is a preamble and the rule text. The preamble is not legally enforceable and varies little between the Proposed and Final Rule, thus it is stored separately from the rule text.
By requirement the SEC publishes all comment letters associated with the publication of a Proposed Rule. The SEC website was programmatically parsed to collect the text of comment letters published for every Proposed Rule between 2000-2021. A dataset of every comment and meeting memorandum was then built (excluding Form Letters) with the associated URL to access to text of comment letter or meeting memorandum as the unique identifier [Following 1999 the SEC began digitizing comment letters submitted and meeting memorandums. They are publicly available online through the SEC website and their unique access URLs served as the unique identifier for each comment or meeting memorandum].
A team of research assistants then manually classified every comment letter and memorandum into one of six categories (see Table 1.1). In addition to manually classification the entity (or entities) represented by the comment letter or meeting were also extracted from the comment (see Appendix 1.1 for description of entity extraction process). In cases where a comment letter was not a Form Letter, but did not represent a larger entity this field was left blank and this comment letter would be subsequently excluded from our analysis (see Table 1.2 for a list of all fields). In cases where a coalition was the author of a comment letter multiple classifications were given as well as multiple entities were extracted.
The text of every comment letter, meeting memorandum, proposed rule, and final rule was then run through a text preprocessing (see Appendix 1.2 for a description of the exact text preprocessing). Text preprocessing was minimal comparative to other potential alternatives in an effort to maintain the true structure of the textual elements. The text of every comment letter, meeting memorandum, proposed rule, and final rule was then parsed for a variety of information (e.g. word count and sentence count). These descriptive factors of each textual element were important to collect given that all comments and rules are not a standardized length. For example, some SEC rules surpass one million words whereas others number in the thousands. Comment letters have similar variation in their length. Additionally, we utilized existing corpora of accounting, finance, and legal terms (jargon) to count the number of occurrences and unique occurrences of these terms within each textual element (see Table 1.3 for a list of all fields) (see Appendix 1.3 for discussion of jargon databases). These fields (e.g. the number of finance terms used) allowed for a better understanding of complexity of rules and comment letters [Generally, it would be expected that more rigorous comment letters would use a more advanced lexicon than those which simply expressed sentiment with regard to a rule].
1.2 Meetings
Records of meetings between the SEC and key stakeholders were also collected. The exact opinions expressed in meetings is not public available, but the rule being discussed is disclosed in addition to other information.
Information on meetings was collected in one of two ways: (1) manual extraction through parsing SEC press releases about meetings with regard to a specific rule (see Appendix 2.1) or (2) meeting memorandums on the Federal Register. These two sources were combined to create an expansive dataset of meetings held by the SEC with key stakeholders. Meetings are distinct from comment letters in that the content of the meetings is not publicly accessible. In many cases the meetings entity also submitted a comment letter to the SEC on the same rule which the meeting was held on. These are believed to closely mirror the opinions expressed in meetings and potentially serve as evidence within lawsuits filed by the stakeholder to demonstrate that a “material” concern was not adequately answered by the SEC before the adoption of a Final Rule.
Meetings were manually classified and entities extracted in same manner as comment letters (see Appendix 1.1 and Table 1.1). Contingent on the source of the meetings addition data was available (see Table 2.1 for a list of all fields). Because meeting memorandums do not contain valuable textual information, with regard to the opinions expressed by the stakeholders, textual analysis was not used on available memorandums.
1.3 Financial Data
Financial data from WRDS, Federal Reserve, and the IRS was manually associated (or matched) with entities that have sent comment letters or met with the SEC at least once between 2000-2021.
Through the SEC’s EDGAR API bulk data on filing companies in the United States is available including their name and Central Index Key (CIK) [https://www.sec.gov/edgar/sec-api-documentation]. A unique CIK is assigned to every firm or individual that files financial disclosure documents with the SEC and does not vary once assigned [https://www.sec.gov/page/edgar-how-do-i-look-central-index-key-cik-number]. All entity names which were manually extracted from the comment letters were then searched for within the set of names of filers in the EDGAR data (see Appendix 1.4 for description of this process). If a match was made than the CIK of the filing/commenting entity was connected to the comments data. Following the association of extracted entity names with CIKs data from Compustat through WRDS was downloaded specifically quarterly market value, share price, and shares outstanding. In addition to other unique identification numbers WRDS also provides a traded firm’s CIK which enabled the connection of financial data to commenting entity which were publicly traded.
IRS-990 data was approached similarly to the WRDS data. Bulk IRS-990 filing data was downloaded including total assets at year end and total yearly expenses [3]. Filing companies reported a firm name was searched in the same manner as described in Appendix 1.3. Following this association process firm which filed with the IRS as non-profit entities and commented/met with the SEC had associated financial data. Finally, the Federal Reserve was used for bank asset under management data (the Federal Reserve limits this to banks over $300 million in AUM) [4]. Using a manual search process (without the process described in Appendix 1.3) bank names were associated with commenting entities were appropriate.
All financial data sources were adjusted for inflation using U.S. Federal Reserve data with 2012 as the base year [5]. Inflation adjustment from the Federal Reserve is calculated from major spending sources within the United States, and their growth overtime reported quarterly and annually. Following adjustment all three datasets could be subset to only included firms which had, at some point in time, commented to or met with the SEC. For the purposes of our analysis only the time period (whether it be quarter or year) during their interaction with the SEC was kept. In cases where data was not available for a given firm during this time period the period closest to their interaction with the SEC following this date was selected. This was done to prevent unnecessary reduction in data availability given that we would expect financial information to exhibit a high degree of autoregression [6].
Appendix
[Appendix 1.1] Entity name extraction process
Comment letters and meeting memorandums both are released with "headers". Headers which do not have an entity named are assumed to not be representing an organization or institution. The proposed rule, "Amendments to Rule 2-01, Qualifications of Accountants" received \# comment letters and \# meeting memorandums. An example of a comment letter header with named entity present is, "Barbara Roper, Director of Investor Protection, Consumer Federation of America". An example of a comment letter header without a named entity present is, "Bill Parrett". In the latter case it is clear that the individual commenting is not representing an organization, whereas in the first comment there is a clear association with the Consumer Federation of America.
The entity extraction process was done manually by a team of research assistants. This was completed manually because the names of entities across comment letter and memorandum headers are not uniform. Thus, there is no sufficiently accurate method to extract entity names. The structure of headers provides additional complication. The formatting of headers with a named entity various as well; in the same proposed rule the following comment header is present, "Ernst and Young LLP". This comment header does not report a named individual commenting nor a date of comment rather just the commenting entity. To sufficiently accurately extract entity names manual extraction was necessary to address the grammatical complexity of the comment and memorandum headers.
Manual extraction was also required in the cases of entities commenting/meeting together. In this case multiple individuals and named entities are reported in the header without (necessarily) a clear distinction.
[Appendix 1.2] Text preprocessing process for textual elements
Text was initially parsed using the SpaCy "en\_core\_web\_sm" pipeline which is built on a variety of text from blogs, news, and comments.\footnote{https://spacy.io/models/en} Following the SpaCy pipeline the text is tokenized by word and by sentence. Word tokens are cleaned of the following characters: ".", ",", ":", ";", "(", ")", "[", "]", and "-". These characters were often split into separate word tokens thus including them in the analysis distorts the focus on textual changes of the document. Our analysis focuses on finding the textual similarity between comment letters and rules. Our analysis focuses on finding textual similarity between comment letters and final rules and it is plausible that even though the words may be the same punctuation is changed to better fit the syntax of a rule.
[Appendix 1.3] Technical language corpora used
The corpora of accounting, finance, and legal words comes from LexPredict’s open-sourced repository of technical words [https://github.com/LexPredict/lexpredict-legal-dictionary, https://lexpredict.com/]. The term corpora are built by scraping key documents and recording the top occurrences. For example, accounting words come from analysis of U.S. GAAP and U.S. FASB whereas legal terms are identified using Black’s 1910 Law Dictionary.