[Wicket-ja-user 287] Re: tomcat clusteringについて

Back to archive index

Takayoshi Kimura takay****@gmail*****
2008年 12月 18日 (木) 00:26:35 JST


その通り、レプリケーションの仕組みの違いです。動作する理由は簡単な
話で、JBossではリクエストを処理するスレッド上、つまりWicketのコン
テキスト内でデシリアライズが行われるからです。ただし、タイミングに
よっては、つまりWicketのフィルタやらコントローラサーブレットに処理
が渡る前に、アプリケーションで設定しているフィルタなどによってセッ
ションへのアクセスが発生してデシリアライズする場合はやっぱり動かな
いかなぁとも思います。

Tomcatのクラスタリングでは、セッションレプリケーションのバックアッ
プ先では受け取ってすぐに、そのオブジェクトが使われるかどうかに関係
なく、バックグラウンドスレッドでオブジェクトを先にデシリアライズす
るようですが、一方JBossは、受け取ったセッションをバイト配列のまま
保持しており、セッションを必要とするリクエストが到達するまでコスト
の高いデシリアライズ処理を行わないようになっています。

Regards,
Takayoshi

2008/12/17 RYO <exten****@gmail*****>:
> 木村さんありがとうございます。
> Wicketのバグということで理解しました。バグとして当現象があがってい
> ないか、確認してみます。
>
> ただ、一点確認させて下さい。
> Jboss asでは動作するというのは、なぜでしょうか?
>
> セッションレプリケーションの仕組みの違いですか?
>
> Jbossは、全然さわったことがないので、ちょっと見当がつかなかったの
> で、もし良ければ教えてください。
>
>
> On 2008/12/17, at 18:15, "Takayoshi Kimura" <takay****@gmail*****> wrote:
>
>> 面白いエラーですね。
>>
>> 原因はHttpSessionに格納されるWicketのクラスの復元処
>> 理にコンテキス
>> ト依存というかスレッド依存の部分がある(Application.get())
>> ためです。
>> 結果的にレプリケーションは失敗してます。Wicketにバグ登録
>> (カテゴリ
>> はバグというよりエンハンスメントかな)するのが良いでしょう
>> ね。
>>
>> JBoss ASならそのまんまで動作すると思います :-)
>>
>> Regards,
>> Yet another Kimura
>>
>> 2008/12/17 木村亮 <exten****@gmail*****>:
>>> みなさん。はじめまして、ryo といいます。
>>>
>>> はじめて、投稿させていただきます。よろしくお願いします。
>>>
>>> wicketを利用してアプリケーション開発を行っていて、
>>> どうしてもうまくない現象がおこってしまいます。質問させてくだ
>>> さい。
>>>
>>> [アプリの大まかな構成]
>>> ・wicket 1.4-m3
>>> ・google-guice 1.0
>>> ・quartz (job sched) 1.6.2
>>> [サーバ構成]
>>> ・apache
>>> ・mod_jk
>>> ・tomcat-5.5.23 (options -Xms256m -Xmx512 -XX:PermSize=64m
>>> -XX:MaxPermSize=128m)
>>>    ・(tomcat物理サーバが3台、バランシングはしてお
>>> らず、
>>>    セッションレプリケーション(SimpleTcpClusterを利用)
>>> のみを行う。スティッキーセッションは無効です。)
>>>    ・同一tomcatインスタンスには、他にも5つ6つ
>>> サービスがのっています。
>>>    ・tomcat server.xmlは以下のような感じです。
>>>http://cwiki.apache.org/WICKET/tomcat-clustering.html
>>> に記述してあるCluster定義とほぼ、一緒です。
>>> 異なる点はSenderのreplicationModeがsynchronous
>>> ではなく、pooledを指定、expireSessionsOnShutdown
>>> を"true"にしています。
>>>
>>> <Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"
>>> managerClassName="org.apache.catalina.cluster.session.DeltaManager"
>>> expireSessionsOnShutdown="true"
>>> useDirtyFlag="true">
>>>
>>> <Membership
>>> className="org.apache.catalina.cluster.mcast.McastService"
>>> mcastAddr="XXXXXXXX"
>>> mcastPort="XXXXXXXX"
>>> mcastFrequency="500"
>>> mcastDropTime="3000"/>
>>>
>>> <Receiver
>>> className="org.apache.catalina.cluster.tcp.ReplicationListener"
>>> tcpListenAddress="auto" ← /etc/hostsをみにいく
>>> tcpListenPort="XXXXXXX"
>>> tcpSelectorTimeout="100"
>>> tcpThreadCount="2"/>
>>>
>>> <Sender
>>> className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"
>>> replicationMode="pooled"/>
>>>
>>> <Valve
>>> className="org.apache.catalina.cluster.tcp.ReplicationValve"
>>> filter=".*\.gif;.*\.js;.*\.jpg;.*\.htm;.*\.html;.*\.txt;.*\.css"/>
>>> </Cluster>
>>>
>>> 以上の状況で、アプリにリクエストを送ると、以下のような例外が吐
>>> かれます。
>>> 2008/12/16 19:26:04 org.apache.catalina.cluster.session.DeltaManager
>>> messageReceived
>>> 致命的: Manager [/context]: Unable to receive message through
>>> TCP channel
>>> java.lang.IllegalArgumentException: Unknown object type null
>>>        at
>>> org.
>>> apache.
>>> wicket.
>>> protocol.
>>> http.
>>> pagestore.
>>> DiskPageStore.restoreAfterSerialization(DiskPageStore.java:1215)
>>>        at
>>> org.apache.wicket.protocol.http.SecondLevelCacheSessionStore
>>> $SecondLevelCachePageMap.
>>> readObject(SecondLevelCacheSessionStore.java:
>>> 406)flect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at
>>> sun.
>>> reflect.
>>> NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at
>>> sun.
>>> reflect.
>>> DelegatingMethodAccessorImpl.
>>> invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:585)
>>>        at
>>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:
>>> 946)
>>>        at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1809)
>>>        at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
>>> 1719)
>>>        at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
>>>        at
>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>>>        at
>>> org.apache.catalina.cluster.session.DeltaRequest
>>> $AttributeInfo.readExternal(DeltaRequest.java:335)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaRequest.readExternal(DeltaRequest.java:247)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.loadDeltaRequest(DeltaManager.java:698)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:
>>> 1573)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.messageReceived(DeltaManager.java:1523)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.messageDataReceived(DeltaManager.java:
>>> 1272)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.
>>> session.
>>> ClusterSessionListener.messageReceived(ClusterSessionListener.java:
>>> 86)
>>>        at
>>> org.
>>> apache.
>>> catalina.cluster.tcp.SimpleTcpCluster.receive(SimpleTcpCluster.java:
>>> 1168)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.
>>> tcp.
>>> ClusterReceiverBase.messageDataReceived(ClusterReceiverBase.java:427)
>>>        at
>>> org.
>>> apache.catalina.cluster.io.ObjectReader.execute(ObjectReader.java:
>>> 108)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.
>>> tcp.TcpReplicationThread.drainChannel(TcpReplicationThread.java:139)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.tcp.TcpReplicationThread.run(TcpReplicationThread.java:70)
>>> Exception in thread
>>> "org.apache.catalina.cluster.tcp.TcpReplicationThread[1]"
>>> java.lang.ExceptionInInitializerError
>>>        at sun.misc.Unsafe.ensureClassInitialized(Native Method)
>>>        at
>>> sun.
>>> reflect.
>>> UnsafeFieldAccessorFactory.
>>> newFieldAccessor(UnsafeFieldAccessorFactory.java:25)
>>>        at
>>> sun.
>>> reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:
>>> 122)
>>>        at java.lang.reflect.Field.acquireFieldAccessor(Field.java:
>>> 917)
>>>        at java.lang.reflect.Field.getFieldAccessor(Field.java:898)
>>>        at java.lang.reflect.Field.getLong(Field.java:527)
>>>        at
>>> java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:
>>> 1586)
>>>        at java.io.ObjectStreamClass.access
>>> $700(ObjectStreamClass.java:52)
>>>        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:408)
>>>        at java.security.AccessController.doPrivileged(Native Method)
>>>        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:
>>> 400)
>>>        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:
>>> 297)
>>>        at
>>> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:531)
>>>        at
>>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:
>>> 1552)
>>>        at
>>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1466)
>>>        at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
>>> 1699)
>>>        at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
>>>        at
>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>>>        at
>>> org.apache.wicket.protocol.http.SecondLevelCacheSessionStore
>>> $SecondLevelCachePageMap.
>>> readObject(SecondLevelCacheSessionStore.java:403)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at
>>> sun.
>>> reflect.
>>> NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at
>>> sun.
>>> reflect.
>>> DelegatingMethodAccessorImpl.
>>> invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:585)
>>>        at
>>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:
>>> 946)
>>>        at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1809)
>>>        at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
>>> 1719)
>>>        at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
>>>        at
>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>>>        at
>>> org.apache.catalina.cluster.session.DeltaRequest
>>> $AttributeInfo.readExternal(DeltaRequest.java:335)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaRequest.readExternal(DeltaRequest.java:247)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.loadDeltaRequest(DeltaManager.java:698)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:
>>> 1573)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.messageReceived(DeltaManager.java:1523)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.messageDataReceived(DeltaManager.java:
>>> 1272)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.
>>> session.
>>> ClusterSessionListener.messageReceived(ClusterSessionListener.java:
>>> 86)
>>>        at
>>> org.
>>> apache.
>>> catalina.cluster.tcp.SimpleTcpCluster.receive(SimpleTcpCluster.java:
>>> 1168)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.
>>> tcp.
>>> ClusterReceiverBase.messageDataReceived(ClusterReceiverBase.java:427)
>>>        at
>>> org.
>>> apache.catalina.cluster.io.ObjectReader.execute(ObjectReader.java:
>>> 108)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.
>>> tcp.TcpReplicationThread.drainChannel(TcpReplicationThread.java:139)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.tcp.TcpReplicationThread.run(TcpReplicationThread.java:70)
>>> Caused by: org.apache.wicket.WicketRuntimeException: There is no
>>> application
>>> attached to current thread
>>> org.apache.catalina.cluster.tcp.TcpReplicationThread[1]
>>>        at org.apache.wicket.Application.get(Application.java:177)
>>>        at org.apache.wicket.Component.getApplication(Component.java:
>>> 1277)
>>>        at org.apache.wicket.Component.<init>(Component.java:893)
>>>        at
>>> org.apache.wicket.MarkupContainer.<init>(MarkupContainer.java:108)
>>>        at org.apache.wicket.Page.<init>(Page.java:238)
>>>        at
>>> org.apache.wicket.protocol.http.pagestore.SerializedPagesCache
>>> $SerializedPageWithSession$1.<init>(SerializedPagesCache.java:206)
>>>        at
>>> org.apache.wicket.protocol.http.pagestore.SerializedPagesCache
>>> $SerializedPageWithSession.<clinit>(SerializedPagesCache.java:205)
>>>        ... 40 more
>>> Exception in thread
>>> "org.apache.catalina.cluster.tcp.TcpReplicationThread[2]"
>>> java.lang.ExceptionInInitializerError
>>>        at sun.misc.Unsafe.ensureClassInitialized(Native Method)
>>>        at
>>> sun.
>>> reflect.
>>> UnsafeFieldAccessorFactory.
>>> newFieldAccessor(UnsafeFieldAccessorFactory.java:25)
>>>        at
>>> sun.
>>> reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:
>>> 122)
>>>        at java.lang.reflect.Field.acquireFieldAccessor(Field.java:
>>> 917)
>>>        at java.lang.reflect.Field.getFieldAccessor(Field.java:898)
>>>        at java.lang.reflect.Field.getLong(Field.java:527)
>>>        at
>>> java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:
>>> 1586)
>>>        at java.io.ObjectStreamClass.access
>>> $700(ObjectStreamClass.java:52)
>>>        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:408)
>>>        at java.security.AccessController.doPrivileged(Native Method)
>>>        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:
>>> 400)
>>>        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:
>>> 297)
>>>        at
>>> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:531)
>>>        at
>>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:
>>> 1552)
>>>        at
>>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1466)
>>>        at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
>>> 1699)
>>>        at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
>>>        at
>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>>>        at
>>> org.apache.wicket.protocol.http.SecondLevelCacheSessionStore
>>> $SecondLevelCachePageMap.
>>> readObject(SecondLevelCacheSessionStore.java:403)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at
>>> sun.
>>> reflect.
>>> NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>        at
>>> sun.
>>> reflect.
>>> DelegatingMethodAccessorImpl.
>>> invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:585)
>>>        at
>>> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:
>>> 946)
>>>        at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1809)
>>>        at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:
>>> 1719)
>>>        at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305)
>>>        at
>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
>>>        at
>>> org.apache.catalina.cluster.session.DeltaRequest
>>> $AttributeInfo.readExternal(DeltaRequest.java:335)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaRequest.readExternal(DeltaRequest.java:247)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.loadDeltaRequest(DeltaManager.java:698)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.handleSESSION_DELTA(DeltaManager.java:
>>> 1573)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.messageReceived(DeltaManager.java:1523)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.session.DeltaManager.messageDataReceived(DeltaManager.java:
>>> 1272)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.
>>> session.
>>> ClusterSessionListener.messageReceived(ClusterSessionListener.java:
>>> 86)
>>>        at
>>> org.
>>> apache.
>>> catalina.cluster.tcp.SimpleTcpCluster.receive(SimpleTcpCluster.java:
>>> 1168)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.
>>> tcp.
>>> ClusterReceiverBase.messageDataReceived(ClusterReceiverBase.java:427)
>>>        at
>>> org.
>>> apache.catalina.cluster.io.ObjectReader.execute(ObjectReader.java:
>>> 108)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.
>>> tcp.TcpReplicationThread.drainChannel(TcpReplicationThread.java:139)
>>>        at
>>> org.
>>> apache.
>>> catalina.
>>> cluster.tcp.TcpReplicationThread.run(TcpReplicationThread.java:70)
>>>
>>> Caused by: org.apache.wicket.WicketRuntimeException: There is no
>>> application
>>> attached to current thread
>>> org.apache.catalina.cluster.tcp.TcpReplicationThread[1]
>>>        at org.apache.wicket.Application.get(Application.java:177)
>>>        at org.apache.wicket.Component.getApplication(Component.java:
>>> 1277)
>>>        at org.apache.wicket.Component.<init>(Component.java:893)
>>>        at
>>> org.apache.wicket.MarkupContainer.<init>(MarkupContainer.java:108)
>>>        at org.apache.wicket.Page.<init>(Page.java:238)
>>>        at
>>> org.apache.wicket.protocol.http.pagestore.SerializedPagesCache
>>> $SerializedPageWithSession$1.<init>(SerializedPagesCache.java:206)
>>>        at
>>> org.apache.wicket.protocol.http.pagestore.SerializedPagesCache
>>> $SerializedPageWithSession.<clinit>(SerializedPagesCache.java:205)
>>>        ... 40
>>> moreorg.apache.catalina.cluster.tcp.TcpReplicationThread
>>>
>>> このスタックトレースを見る限りだ
>>> と、org.apache.catalina.cluster.tcp.TcpReplicationThread
>>> のThreadLocalからApplicationオブジェクトが取得でき
>>> ずに、
>>> WicketRuntimeExceptionをはいています。
>>>
>>> これは、単純に、セッションレプリケーションができていないとみな
>>> してよいでしょうか?
>>> tomcatの一時領域には、disk page storeファイルが出力されて
>>> いました。(1台のみ)
>>>
>>> スタックトレースの
>>> at
>>> org.
>>> apache.
>>> wicket.
>>> protocol.
>>> http.
>>> pagestore.
>>> DiskPageStore.restoreAfterSerialization(DiskPageStore.java:1215)
>>> のソースを追ってみると、SecondLevelCacheSessionStore
>>> $SecondLevelCachePageMap
>>> クラスの以下の箇所で、IllegalArgumentExceptionが投げられ
>>> ていました。
>>>
>>> private void readObject(java.io.ObjectInputStream s) throws
>>> IOException,
>>>            ClassNotFoundException
>>>        {
>>>            s.defaultReadObject();
>>>
>>>            IPageStore store = getPageStore();
>>>
>>>            if (sessionId != null && store instanceof
>>> IClusteredPageStore ==
>>> false)
>>>            {
>>>                Object lastPage = s.readObject();
>>>                if (store instanceof ISerializationAwarePageStore)
>>>                {
>>>                    lastPage =
>>> (
>>> (
>>> ISerializationAwarePageStore
>>> )store).restoreAfterSerialization((Serializable)lastPage);
>>> ←ここ
>>>                }
>>>                this.lastPage = lastPage;
>>>            }
>>>        }
>>>
>>> DiskPageStore.restoreAfterSerializationメソッド、
>>>
>>> /**
>>>     * @see
>>> org.
>>> apache.
>>> wicket.
>>> protocol.
>>> http.
>>> SecondLevelCacheSessionStore.
>>> ISerializationAwarePageStore#
>>> restoreAfterSerialization(java.io.Serializable)
>>>     */
>>>    public Object restoreAfterSerialization(Serializable serializable)
>>>    {
>>>        if (!storeAfterSessionReplication() || serializable
>>> instanceof Page)
>>>        {
>>>            return serializable;
>>>        }
>>>        else if (serializable instanceof SerializedPageWithSession)
>>>        {
>>>            SerializedPageWithSession page =
>>> (SerializedPageWithSession)serializable;
>>>            if (page.page == null || page.page.get() !=
>>> SerializedPageWithSession.NO_PAGE)
>>>            {
>>>                storeSerializedPages(page.sessionId, page.pages);
>>>                return stripSerializedPage(page);
>>>            }
>>>            else
>>>            {
>>>                return page;
>>>            }
>>>        }
>>>        else
>>>        {
>>>            String type = serializable != null ?
>>> serializable.getClass().getName() : null;
>>>            throw new IllegalArgumentException("Unknown object type
>>> " +
>>> type); ←ここでIllegalArgumentExceptionがスロー
>>>        }
>>>    }
>>> 引数に渡されるserializable(SecondLevelCacheSessionStore
>>> $SecondLevelCachePageMap#readObject(ObjectInputStreamから読み込
>>> んだlastPage)がnullのため、IllegalArgumentException
>>> をスローしていました。
>>> なぜ、このような状況になるのかの原因の切り分けが現状できていま
>>> せん。
>>>
>>> 何か、参考になるようなことがあれば、ご教授いただきたく投稿させ
>>> ていただきました。
>>> 以上よろしくお願いいたします。
>>>
>>> _______________________________________________
>>> Wicket-ja-user mailing list
>>> Wicke****@lists*****
>>> http://lists.sourceforge.jp/mailman/listinfo/wicket-ja-user
>>>
>>>
>>
>> _______________________________________________
>> Wicket-ja-user mailing list
>> Wicke****@lists*****
>> http://lists.sourceforge.jp/mailman/listinfo/wicket-ja-user
>> _______________________________________________
>>> Wicket-ja-user mailing list
>>> Wicke****@lists*****
>>> http://lists.sourceforge.jp/mailman/listinfo/wicket-ja-user
>>>
>>>
>>
>> _______________________________________________
>> Wicket-ja-user mailing list
>> Wicke****@lists*****
>> http://lists.sourceforge.jp/mailman/listinfo/wicket-ja-user
>> t-ja-user
>>
>
> _______________________________________________
> Wicket-ja-user mailing list
> Wicke****@lists*****
> http://lists.sourceforge.jp/mailman/listinfo/wicket-ja-user
>



Wicket-ja-user メーリングリストの案内
Back to archive index