Development - Various Information

Table Of Contents:

Contents

Introduction
How to contribute with patches to NetarchiveSuite
Developer Short Cuts
Repository
Projects in pipeline
Current Project Information
Development Methodology
1. Processes
2. Roles
Guidelines & Policies
Generated reports

Introduction

The NetarchiveSuite system is continuously under development. The information given on this page is primarily information used in development work, including coordination between different institutions with developers.

The Communication section gives information of to communicate as a developer or contributer.

The Developer Short Cuts section lists usefull short cuts when doing development work, however it may be a good idea to get acquainted with repository, methodology and guidelines & policies before using these short cuts.

The Repository section explains where code and code related information is placed.

The Projects in pipeline section gives reference to various design discussion and assignments for tasks that may or may not become implemented at a later stage. This is mainly concerned with bigger changes.

The Current Project Information contains the current and past information about iterations, produced as part of following the development methodology as well as the current staffing of the roles used in the development methodology.

The Development Methodology section describes the processes and related roles which is basis for how the development work is done.

The Guidelines & Policies section includes references to guidelines that supports various processes and administration routines.

The Generated reports section includes references to automatically generated reports, mainly concerned with code and bugsthat supports.

How to contribute with patches to NetarchiveSuite

edit

Description of how to contribute is given by the process description (especially the Tracker Issue Life-Cycle process which includes an Implementation process) and various coding guidelines (including unit test). If you read it and follow the suggestions therein, you will not only make it easier to get your changes to NetarchiveSuite be accepted into the main distribution, you will also likely improve the quality of your code overall.

We're always happy to receive patches, though we may choose not to apply them if the implemented features go against our purposes or the code quality is too low.

If possible, use our patch-tracker to transmit the patch to us. The Create Tracker Issue process describes how.

If sending patches by email, please follow the Create Tracker Issue process as far as possible, and please send files as attachments rather than inline, as mail readers tend to mess up important formatting.

For further information please to the Communication tab.

Developer Short Cuts

edit

Shortcuts are for developers and will reflect the contents of this page, but it will also include links to items on other wikis that are not yet public. To link to this place use the link http://netarchive.dk/suite/Development#DeveloperShortCuts.

Repository

edit

To follow the latest developments, go to the SVN repository, which contains latest updates of code and see the current list of bugs, patches and feature requests.

The guide for how to register a bug, a patch or a feature request can be found at Bug/Feature Request/Patch Information Guideline

But check first, that the bug or feature request has not been reported already: list of existing bugs, patches and feature requests

To see the latest releases go to registration and download

Projects in pipeline

edit

Assignments

Assignments have not yet been generally moved to from our internal wiki to this wiki, but new assignments will show up here until we get the rest moved. The assignments are divided by module.

For further description of the modules, please refer to Module description in System Design document.

For further descripttion of naming conventions of assignments, please refer to the Assignment Description Guidelines in section Guidelines & Policies.

Assignments for module Access

Access Assignment 1 Develop working prototype for Wayback access
(older version: WaybackAssignments20081119.odt )
- Configuration of Wayback Test System
- Configuration of Wayback Prototype (beta-version)
Access Assignment 2 Integrate Wayback development into main project
- Integrate Wayback Development into Main Project
- Tools for Generation of Wayback-ready Indexes - Specifically Deduplication and Correct HTTP-return codes
Access Assignment 3 Tools for Automatic Maintenance of Wayback Installation
Access Assignment 4 NutchWAX integration into NetarchiveSuite.

Assignments for module Archive

Assignment Group B.2 Improve and isolate bit preservation GUI.
Assignment Group B.4 Improve batch job architecture

Assignments for module Common

There are currently no new assignments for the Common module.

Assignments for module Deploy

Deploy Assignment 1 Rewrite deploy based on settings overwrite (DONE)

Assignments for module Harvester

Harvester Assignment 1 Upgrade to Heritrix release 3
Harvester Assignment 2 Move to WARC writing instead of ARC writing
Harvester Assignment 3 A tool for monitoring broad crawls
Harvester Assignment 4 Global Crawler Traps

Assignments for module Monitor

There are currently no new assignments for the Monitor module.

Old Assignments

These are only represented on the old wiki:

Requirement list

These only exist on the old Wiki as requirement list. There are in the process of being put into Feature Requests.

Current Project Information

edit

Staffing

Staffing of Roles

Role (Updated 17. February 2010)	Responsible	Backup
Documentation Coordinator	TLR	JOLF
External Software Coordinator	SVC	TLR
Internal Administrator	SVC	CSR
Project Leader	CHH	CSR
Quality Assurance Coordinator	SVC	CSR
Test Coordinator	TLR	Quality Assurance Coordinator

Staffing of Module Ownership

Main Module (Updated 17. February 2010)	Module Owner	Backup
Access	CSR	Quality Assurance Coordinator
Archive*	JOLF	Quality Assurance Coordinator
Common	CSR	Quality Assurance Coordinator
Harvester GUI part	SVC	CSR
Harvester harvest part	SVC	CSR
Deploy	JOLF	Quality Assurance Coordinator
Other Module	Module Owner	Backup
Documentation	JOLF	Quality Assurance Coordinator
Monitor	CSR	Quality Assurance Coordinator
Test	TLR	Quality Assurance Coordinator

* Archive includes: arcrepository, batch, bitarchive (and checksum archive), bitpreservation, logicalpreservation, Indexserver.

New staffing proposal Feb 2010

Current Iteration

Please refer to Iteration 42

Old Iterations (incl. Test, reviews)

Iteration 41: Ended February 18 2010 with the release of NetarchiveSuite 3.11.2

Iteration 40: Ended December 18 2009 with the release of NetarchiveSuite 3.11.0

Iteration 39: Ended November 16 2009 with the release of NetarchiveSuite 3.10.0

Iteration 38: Ended October 06 2009 with the release of NetarchiveSuite 3.9.2

Iteration 37: Ended August 10 2009 with the release of NetarchiveSuite 3.9.0

Iteration 36: Ended May 25 2009 with the release of NetarchiveSuite 3.8.0

For Iterations before 36, please refer to old iterations on old wiki (not open).

Development Methodology

edit

The development methodology described here is inspired by CMMI (http://www.sei.cmu.edu/cmmi/). It will be described in form of processes and roles.

The processes are either main processes or subprocesses covered by a main process. Each main process includes an overview and the processes and roles involved. Each subprocess includes a description, a purpose, the role responsible for the subprocess, the method used in form of activities, the time it takes place and its input and output.

Each role includes a description, the processes the role is responsible for, and the guidelines relevant for the role, any references to current work and backup staffing list.

To insert a new process or role description, please refer to the Guidelines for Writing a new Process or Role.

Processes

NB: Not all processes are translated and transferred from the old wiki - this work is in progress and will primarily be taken from the Current Methodology (not open).

There are two types of processes:

the main processes that themselves consist of processes. An example is Tracker Issue Life-Cycle which contains all processes related to tracker issues like bugs, feature requests and patches.
processes that describe methods, input and output for for example Creation of a Tracker Issue like a bug, a feature request or a patch.

To get the page with all process descriptions in structured form, please refer to All Processes.

Below is first given list of all main processes links, followed by links to all process descriptions in alphabetic order. Main processes are indicated by bold.

All processes in alphabetic order:

Roles

NB: Not all roles are translated and transferred from the old wiki - this work is in progress and will primarily be taken from the Current Methodology (not open).

The methodology refers to different roles like QA, Module Owner, etc. However a few roles are not described, since they in principle are out of scope of the NetarchiveSuite. These are:

Operation Manager
Steering Committee

To get the page with descriptions of all our roles, please refer to All Roles page.

Below is given links to role descriptions in alphabetic order.

Documentation Coordinator
External Software Coordinator
Internal Administrator
Module Owner
Operation Manager
Project Leader
Quality Assurance Coordinator
Steering Committee
Task Holder (Implementer, assignment writer, tester etc.)
Test Coordinator

Guidelines & Policies

edit

New Process

Please refer to the Guidelines for writing a new process or role

Code related

Maintain Documentation

The documentation for the NetarchiveSuite software is maintained for both the current stable branch, and the development branch. The document Maintaining Documentation describes the steps necessary for transferring the development branch documentation to the stable branch, when making a stable release.

To produce PDFs, please refer to the Generate PDF from manual guideline.

Assignment Description Guidelines

Please refer to Assignment Writing

Release Policy

There are several things to do when we make a release. Please see the page on Release Procedures for further information.

Tracker (Bug/Feature-Request/Patch) registration

Please refer to the bug information guideline

Placement of log-files

Please refer to the Hints to Where to Find Log-files

Crucible Guidelines for sign-up

Crucible Administration Guidelines

Generated reports

edit

Currently most generated reports are placed on a server that is not accessible to the outside. This will be changed when the information is moved to a public accessible server.