* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements.  See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership.  The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License.  You may obtain a copy of the License at

*

*   http://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing,

* software distributed under the License is distributed on an

* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

* KIND, either express or implied.  See the License for the

* specific language governing permissions and limitations

* under the License.

*/

* User callable serialization and deserialization of the CAS in a compressed Binary Format

*

* This serializes/deserializes the state of the CAS.  It has the capability to map type systems,

* so the sending and receiving type systems do not have to be the same.

*   - types and features are matched by name, and features must have the same range (slot kind)

*   - types and/or features in one type system not in the other are skipped over

*

* Header specifies to reader the format, and the compression level.

*

* How to Serialize: 

*

* 1) create an instance of this class

*    a) if doing a delta serialization, pass in the mark and a ReuseInfo object that was created

*       after deserializing this CAS initially.

*    b) if serializaing to a target with a different type system, pass the target's type system impl object

*       so the serialization can filter the types for the target.

* 2) call serialize() to serialize the CAS

* 3) If doing serialization to a target from which you expect to receive back a delta CAS,

*    create a ReuseInfo object from this object and reuse it for deserializing the delta CAS.

*   

* TypeSystemImpl objects are lazily augmented by customized TypeInfo instances for each type encountered in

* serializing or deserializing.  These are preserved for future calls, so their setup / initialization is only

* needed the first time.

*

* TypeSystemImpl objects are also lazily augmented by typeMappers for individual different target typesystems;

* these too are preserved and reused on future calls.

*

* Compressed Binary CASes are designed to be "self-describing" -

* The format of the compressed binary CAS, including version info,

* is inserted at the beginning so that a proper deserialization method can be automatically chosen. 

*    

* Compressed Binary format implemented by this class supports type system mapping. 

* Types in the source which are not in the target

* (or vice versa) are omitted. 

*   Types with "extra" features have their extra features omitted

*   (or on deserialization, they are set to their default value - null, or 0, etc.).

*   Feature slots which hold references to types not in the target type system are replaced with 0 (null). 

*

* How to Deserialize:

*

* 1) get an appropriate CAS to deserialize into.  For delta CAS, it does not have to be empty, but it must

*    be the originating CAS from which the delta was produced.

* 2) If the case is one where the target type system == the CAS's, and the serialized for is not Delta,

*    then, call aCAS.reinit(source).  Otherwise, create an instance of this class -> xxx

*    a) Assuming the object being deserialized has a different type system,

*       set the "target" type system to the TypeSystemImpl instance of the

*       object being deserialized.   

*    a) if delta deserializing, pass in the ReuseInfo object created when the CAS was serialized

* 3) call xxx.deserialize(inputStream)

*

* Compression/Decompression

* Works in two stages:

*   application of Zip/Unzip to particular sub-collections of CAS data,

*     grouped according to similar data distribution

*   collection of like kinds of data (to make the zipping more effective)

*   There can be up to ~20 of these collections, such as

*      control info, float-exponents, string chars

* Deserialization:

*   Read all bytes,

*   create separate ByteArrayInputStreams for each segment

*   create appropriate unzip data input streams for these

*  

*   Slow but expensive data:

*     extra type system info - lazily created and added to shared TypeSystemImpl object

*       set up per type actually referenced

*     mapper for type system - lazily created and added to shared TypeSystemImpl object

*       in identity-map cache (size limit = 10 per source type system?) - key is target typesystemimpl.

*   Defaulting:

*     flags:  doMeasurements, compressLevel, CompressStrategy

*   Per serialize call: cas, output, [target ts], [mark for delta]

*   Per deserialize call: cas, input, [target ts], whether-to-save-info-for-delta-serialization

*  

*   CASImpl has instance method with defaulting args for serialization.

*   CASImpl has reinit which works with compressed binary serialization objects

*     if no type mapping

*     If type mapping, (new BinaryCasSerDes6(cas,

*                                            marker-or-null,

*                                            targetTypeSystem (for stream being deserialized),

*                                            reuseInfo-or-null)

*                                                .deserialize(in-stream)

*    

* Use Cases, filtering and delta

*   **************************************************************************

*   * (de)serialize * filter? * delta? * Use case

*   **************************************************************************

*   * serialize     *   N     *   N    * Saving a Cas,

*   *               *         *        * sending Cas to service with identical ts

*   **************************************************************************

*   * serialize     *   Y     *   N    * sending Cas to service with

*   *               *         *        * different ts (a guaranteed subset)

*   **************************************************************************

*   * serialize     *   N     *   Y    * returning Cas to client

*   *               *         *        *   uses info saved when deserializing

*   *               *         *        * (?? saving just a delta to disk??)

*   **************************************************************************

*   * serialize     *   Y     *   Y    * NOT SUPPORTED (not needed) 

*   **************************************************************************

*   * deserialize   *   N     *   N    * reading/(receiving) CAS, identical TS

*   **************************************************************************

*   * deserialize   *   Y     *   N    * reading/receiving CAS, different TS

*   *               *         *        * ts not guaranteed to be superset

*   *               *         *        * for "reading" case.

*   **************************************************************************

*   * deserialize   *   N     *   Y    * receiving CAS, identical TS

*   *               *         *        *   uses info saved when serializing

*   **************************************************************************

*   * deserialize   *   Y     *   Y    * receiving CAS, different TS (tgt a feature subset)

*   *               *         *        *   uses info saved when serializing

*   **************************************************************************

*/

  private static final int[] INT0 = new int[0];

 

  private static final boolean TRACE_SER = false;

  private static final boolean TRACE_DES = false;

 

  private static final boolean TRACE_STR_ARRAY = false;

  /**

   * Version of the serializer/deserializer, used to allow deserialization of

   * older versions

   *

   * Version 0 - initial SVN checkin

   * Version 1 - changes to support CasTypeSystemMapper

   */

  private static final int VERSION = 1; 

 

  private static final long DBL_1 = Double.doubleToLongBits(1D);

  /**

   * Compression alternatives

   */

 

  public enum CompressLevel {

    None(   Deflater.NO_COMPRESSION),

    Fast(   Deflater.BEST_SPEED),

    Default(Deflater.DEFAULT_COMPRESSION),

    Best(   Deflater.BEST_COMPRESSION),

    ;

    final public int lvl;

    CompressLevel(int lvl) {

      this.lvl = lvl;

    }

  }

 

  public enum CompressStrat {

    Default(      Deflater.DEFAULT_STRATEGY),

    Filtered(     Deflater.FILTERED),

    HuffmanOnly(  Deflater.HUFFMAN_ONLY),

    ;

    final public int strat;

    CompressStrat(int strat) {

      this.strat = strat;

    }

  }

  /**

   * Info reused for

   *   1) multiple serializations of same cas to multiple targets (a speedup), or

   *   2) for delta cas serialization, where it represents the fsStartIndex info before any mods

   *      were done which could change that info, or

   *   3) for deserializing with a delta cas, where it represents the fsStartIndex info at the time

   *      the CAS was serialized out..

   * Reachable FSs and Sequence maps

   */

  public static class ReuseInfo {

    /**

     * kept to avoid recomputation in the use case:

     *   - serialize to target 1, serialize same to target 2, etc.

     *   - Delta serialization (uses reuse info saved during initial deserialization)

     *   - Delta deserialization

     *   if Null, recomputed when needed

     * BitSet used to test if fsRef needs to be serialized  

     */

    final private BitSet foundFSs;

    final private int[] foundFSsArray; // ordered set of FSs found in indexes or linked from other found FSs

   

    /**

     * Multiple uses:

     *   a) avoid recomputation when multiple serializations of same CAS to multiple targets

     *   b) remembers required mapping for processing delta cas serializations and deserializations conversion of tgt seq # to src addr

     */

    final private CasSeqAddrMaps fsStartIndexes;

   

    private ReuseInfo(

        BitSet foundFSs,

        int[] foundFSsArray,

        CasSeqAddrMaps fsStartIndexes) {

      this.foundFSs = foundFSs;

      this.foundFSsArray = foundFSsArray;

      this.fsStartIndexes = fsStartIndexes;

    }

  }

 

  public ReuseInfo getReuseInfo() {

    return new ReuseInfo(foundFSs, foundFSsArray, fsStartIndexes);

  }

   

  // speedups - ints for SlotKind ordinals

  final private static int arrayLength_i = Slot_ArrayLength.ordinal();

  final private static int heapRef_i = Slot_HeapRef.ordinal();

  final private static int int_i = Slot_Int.ordinal();

  final private static int byte_i = Slot_Byte.ordinal();

  final private static int short_i = Slot_Short.ordinal();

  final private static int typeCode_i = Slot_TypeCode.ordinal();

  final private static int strOffset_i = Slot_StrOffset.ordinal();

  final private static int strLength_i = Slot_StrLength.ordinal();

  final private static int long_High_i = Slot_Long_High.ordinal();

  final private static int long_Low_i = Slot_Long_Low.ordinal();

  final private static int float_Mantissa_Sign_i = Slot_Float_Mantissa_Sign.ordinal();

  final private static int float_Exponent_i = Slot_Float_Exponent.ordinal();

  final private static int double_Mantissa_Sign_i = Slot_Double_Mantissa_Sign.ordinal();

  final private static int double_Exponent_i = Slot_Double_Exponent.ordinal();

  final private static int fsIndexes_i = Slot_FsIndexes.ordinal();

  final private static int strChars_i = Slot_StrChars.ordinal();

  final private static int control_i = Slot_Control.ordinal();

  final private static int strSeg_i = Slot_StrSeg.ordinal();

  /**

   * Things set up for one instance of this class

   */

  private TypeSystemImpl ts;

  final private CompressLevel compressLevel;

  final private CompressStrat compressStrategy; 

 

  /**

   * Things that are used by common routines among serialization and deserialization

   */

  private boolean isTypeMappingCmn;

  private CasTypeSystemMapper typeMapperCmn;

  /*****************************************************

   *  Things for both serialization and Deserialization

   *****************************************************/

  final private CASImpl cas;  // cas being serialized

  private int[] heap;           // main heap, can't be final because grow replaces it

  final private StringHeap stringHeapObj;

  final private LongHeap longHeapObj;

  final private ShortHeap shortHeapObj;

  final private ByteHeap byteHeapObj;

  private int heapStart;

  private int heapEnd;                 // set when deserializing  

  private int totalMappedHeapSize = 0; // heapEnd - heapStart, but with FS that don't exist in the target type system deleted   

  final private boolean isSerializingDelta;        // if true, there is a marker indicating the start spot(s)

        private boolean isDelta;

        private boolean isReadingDelta;

  final private MarkerImpl mark;  // the mark to serialize from

  /**

   *

   */

  final private CasSeqAddrMaps fsStartIndexes;

  final private boolean reuseInfoProvided;

  final private boolean doMeasurements;  // if true, doing measurements